Monday, July 18, 2016

Working with WSO2 RabbitMQ Inbound

I am writing this blog by considering you already have knowledge about inbound and RabbitMQ. If not please refer my previous blogs. It has brief idea about those.

In WSO2 RabbitMQ inbound, we use inbound as RabbitMQ consumer. It is event based inbound which polls only once to establish a connection with the remote server and then consumes events.  In here AMQP messaging protocol is used. AMQP is a wire-level messaging protocol that describes the format of the data that is sent across the network. If a system or application can read and write AMQP, it can exchange messages with any other system or application that understands AMQP regardless of the implementation language.

Steps to enable RabbitMQ Inbound 
  • Start the WSO2 Server by going <ESB_HOME>/bin by specifying this command sh ./wso2server.sh
  • Create  two simple sequences.  In my case I have created simple sequence names as TestIn and amqpErrorSeq with following configuration
    <?xml version="1.0" encoding="UTF-8"?>
    <sequence name="TestIn" onError="amqpErrorSeq" xmlns="http://ws.apache.org/ns/synapse">
        <log level="full"/>
        <drop/>
    </sequence>



    <?xml version="1.0" encoding="UTF-8"?>
    <sequence name="amqpErrorSeq" xmlns="http://ws.apache.org/ns/synapse">
        <log level="full"/>
        <drop/>
    </sequence>
  •  Go to inbound endpoints in Service Bus menu.
  • Click on Add Inbound endpoint. you will see the page like below.

  • Enter the name of the endpoint and select type as rabbitmq. Click on Next.
  • In the next page specify 

    • Sequence as TestIn.
    • Error Sequence as amqpErrorSeq. 
    • Suspend as false. We need to put as false to make inbound in enable mode. if we put true, the inbound is disabled.
    • Sequential as true.
    • Coordination as true.
    • rabbitmq.connection.factory as AMQPConnectionFactory
    • rabbitmq.server.host.name as localhost
    • rabbitmq.server.port as 5672. Here management console of rabbitmq is 15672. But here we have to mention as 5672.
    • rabbitmq.server.user.name as guest (In my case i am using guest account).
    • rabbitmq.server.password as guest (In my case i am using guest account).
    • rabbitmq.queue.name : specify a queue name.
  • rabbitmq.exchange.name : specify exchange name.
  • If you want to change anything in advanced options, click it and change it.
Thats all for the RabbitMq inbound. That means we have created consumer. For consuming events, we need to provide events. so i have created a sample java producer. That was given below. You can see the output of the message in ESB started terminal.

import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

public class Producer {
    private final static String QUEUE_NAME = "queue";
    private final static String contentEncoding = "utf-8";
    public static void main(String[] argv) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        factory.setUsername("guest");
        factory.setPassword("guest");
        factory.setPort(5672);
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.queueDeclare(QUEUE_NAME, false, false, false, null);
        channel.exchangeDeclare("exchange", "direct", true);
        channel.queueBind(QUEUE_NAME, "exchange", "route");
        String message ="<m:placeOrder xmlns:m=\"http://services.samples\">" +
                "<m:order>" +
                "<m:price>100</m:price>" +
                "<m:quantity>20</m:quantity>" +
                "<m:symbol>RMQ</m:symbol>" +
                "</m:order>" +
                "</m:placeOrder>";
        // Populate the AMQP message properties
        AMQP.BasicProperties.Builder builder = new AMQP.BasicProperties().builder();
        builder.contentType("application/xml");
        builder.contentEncoding(contentEncoding);

        // Publish the message to exchange
        channel.basicPublish("exchange", QUEUE_NAME, builder.build(), message.getBytes());
        System.out.println(" [x] Sent '" + message + "'");
        channel.close();
        connection.close();
    }
}

The out is given below.



Wednesday, July 13, 2016

Working with own SOAP Receiver in WSO2 CEP

Hi all,

In this blog we are going to see from the beginning. Here I am doing one example from the start. You can do it same way for your scenario.

  • First we need to create stream. I created stream named as "soapStream". My created stream is given below.

  • Then we need to create a soap event receiver under Manage->Receivers in Management console. I created event receiver names as "soapTest". My created soap event receiver is given below.

  • Then we need to have publisher to view the output to check whether our input is going inside. I created a logger publisher named as "soapTestLogger" to view the output in terminal.

  • Now we have finished our initial steps. As we mentioned transport as http in Soap receiver which we created, we need to give http soap request. Here I used Postman (which is google chrome apps) to send the request. http address we have to use is http://localhost:9763/services/<ReceiverName>/receive. My receiver name is soapTest. so i used like http://localhost:9763/services/soapTest/receive. In header we have to mention Content-Type as application/xml. In body we have to give our events according to the stream. The body part i gave is given below.

    <soap:Envelope
    xmlns:soap="http://www.w3.org/2003/05/soap-envelope/"
    soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding"><soap:Body>
      <events>
        <event>
            <payloadData>
                <ID>70</ID>
                <Name>Test Data</Name>
            </payloadData>
        </event>
    </events>
    </soap:Body>

    </soap:Envelope>
The postman screen shot is given below.


 The output can view in the terminal(CEP started Terminal).

 Now we have finished using soap Receiver with Postman http request. Will see you in next blog. :)

Working with SOAP Receiver Sample in WSO2 CEP

Introduction to WSO2 CEP SOAP Receiver

Events are received by WSO2 CEP server using event receivers, which manage the event retrieval process. WSO2 CEP receives events via multiple transports in JSON, XML, Map, Text, and WSO2Event formats, and converts them into streams of canonical WSO2Events to be processed by the server. SOAP event receiver is used to receive events in XML format via HTTP, HTTPS, and local transports.

Doing the soap sample already available

WSO2 CEP has samples for every Event Receivers it has. You can run the sample very easily like below.
Steps
  • Start the CEP Server sample by going <CEP_HOME>/bin by specifying this command ./wso2cep-samples.sh -sn 0014
    We specify sample number as 0014 because of soap sample receiver is in that. you can view that by going <CEP_HOME>/samples/artifacts/0014/eventreceivers
The receiver is like this.

<?xml version="1.0" encoding="UTF-8"?>
<eventReceiver name="soapReceiver" statistics="disable" trace="disable" xmlns="http://wso2.org/carbon/eventreceiver">
    <from eventAdapterType="soap">
        <property name="transports">all</property>
    </from>
    <mapping customMapping="disable" type="xml"/>
    <to streamName="org.wso2.event.sensor.stream" version="1.0.0"/>
</eventReceiver> 
  • After starting the CEP server, you can see the stream named as "org.wso2.event.sensor.stream" and receiver "soapReceiver" in your CEP management console.
  • As the next step, you need to produce soap events which can be receive by CEP. There is sample java soap producer available in <CEP_HOME>/samples/producers/soap. So you just want to ant build it. For that in terminal, you have to go to above mentioned directory and use below Ant command.

    ant -Durl=http://localhost:9763/services/soapReceiver/receive -Dsn=0014

    Here http://localhost:9763/services/soapReceiver/receive means http location of the receiver. Then we specify the  the sample number. The xml format input data is taken from <CEP_HOME>/samples/artifacts/0014/soapReceiver.txt
  • After this ant command you can see the output in the terminal of CEP started like below.

 Thats All, now you have successfully ran the sample. In my next blog, we will focus on creating own Soap receiver, stream and sending soap request to that. Keep in touch :)




Friday, July 1, 2016

Installing RabbitMQ in Ubunthu

Introduction to RabbitMQ

Initially when I search in Internet i found that RabbitMQ is broker. I also could not understand what they meant by that. After some researches only I could understand. RabbitMQ task is getting the messages and forward it to the receiver. I am sure you are familiar with post office system. In post office post man will get the messages from post box and he will give it to post office. In post office we categories the posts according to the regions and deliver it to relevant places through the postman. This is the procedure of RabbitMQ also. RabbitMQ will act as postman, post office and post box. The thing is post office deals with papers and RabbitMQ deals with blobs of data. A program that sends messages is called as "Producer"(P). Queue is the one which stores messages for a sometime. It is with in RabbitMQ. We can say queue is like post box. A queue is not bound by any limits, it can store as many messages as you like ‒ it's essentially an infinite buffer. Like humans put posts into the post box, a producer can send messages to queue. In this case many consumers can try to receive data from one queue. A "Consumer (C)" is a program that mostly waits to receive messages. Note that the producer, consumer, and broker do not have to reside on the same machine; indeed in most applications they don't. We can write producer and consumer in different languages.

Steps to Install RabbitMQ
  1. You have to check whether RabbitMQ is already in your Ubunthu OS. You can use this code in your terminal.
    sudo rabbitmq-server start

    If yo u get message like "node with name "rabbit" already running on ", then you can know that in your OS, RabbitMQ is already installed. But you can upgrade new version. To upgrade/newly install, you can use following steps.
  2. Execute the following command to add the APT repository.
     sudo add-apt-repository "deb http://www.rabbitmq.com/debian/ testing main"
  3. To avoid warnings about unsigned packages, add our public key to your trusted key list using below command.
    wget https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
  4. After the above command, use this command.
    sudo apt-key add rabbitmq-signing-key-public.asc
  5. Run the following command to update the package list.
    sudo apt-get update
  6.  Install rabbitmq-server package
    sudo apt-get install rabbitmq-server
    In this time, if you had RabbitMQ already, then you could upgrade the version of it. If you don't have it already, new version have installed at this time and start the server automatically.
  7.  You can check whether it start/not by following command.
    sudo rabbitmqctl status
  8. In browser you can type like this http://localhost:15672/In this time if you don't get interface like Figure 1, you have to activate plugins (mentioned in step 9).
  9. Enable plugins.
    sudo rabbitmq-plugins enable rabbitmq_management
  10.  You can use "guest" as username and password.
  11. Yes. I have successfully installed RabbitMQ now. :) In my next blogs, we will focus more on RabbitMQ.

Figure 1






Wednesday, June 29, 2016

Siddhi Extension- Window Extension

Hi all,

Introduction to WSO2 CEP

WSO2 Complex Event Processor (CEP) is a lightweight, easy-to-use, open source Complex Event Processing server (CEP). CEP is used to real time processing of the data. We can give data through the receiver (There are lot of ways to give data to CEP such as HTTP format, Kafka, WSO2 event..), process the stream by writing siddhi queries in execution plan and publish it in different format.

According to the siddhi queries we can process the data which is coming from receiver and publish it. So Siddhi core is the part doing process of data which is more important in real time processing.

Introduction to Siddhi

Siddhi Query Language (SiddhiQL) is designed to process event streams to identify complex event occurrences. We have to add streams for input and output to write execution plans and siddhi queries. the queries is applicable to that stream.

Creating stream for Input
  • Start the CEP server using sh ./wso2server.sh command in terminal
  • Go to https://localhost:9443/carbon
  • In the Management Console, under Manage tag you could see Streams. Click on that.
  • Then you click Add Event Stream. Then you will see following window.

  • In that Specify the name, version, description and Attributes. The data should be given in the order of attributes.
  • Like that define your output stream also.
Writing queries in Execution Plans
  •  Click on Execution plans under the Streaming Analytics in Manage and click on Add New Execution Plan.
  • You could see below window.
  • In that you can import stream (that means your input stream related to receiver) and export stream (that means your output stream related to publisher)
  • After that you could write your siddhi queries in that by referring streams and you can put result in output stream. 
Window Siddhi Extension

Window extension is type of siddhi extension.  Window Extension allows events to be collected and expired without altering the event format based on the given input parameters like the Window operator You can get the archetype of the window extension in here. After cloning this git repo and follow setup.txt file to get your sample project. Specify the class name as your wish when you are creating project from archetype. Then the main class and test case class will be created with wanted classes. The structure of the project is given below.




Here we will see the purposes of methods in SampleWindow.java class.
  • init : This method will be called before other methods. When we writing queries we write parameter value for method.
    For e.g window.unique:length(ip,3). This init method will take that parameter values from the query.
  • process : The main processing method that will be called upon event arrival. The logic of the window extension will be written here. We have to write what kind of processes we have to apply for arriving event.
  • find : To find events from the processor event pool, that the matches the matching Event based on finder logic.
  • constructFinder : To construct a finder having the capability of finding events at the processor that corresponds to the incoming matching Event and the given matching expression logic.
  • start : This will be called only once and this can be used to acquire required resources for the processing element.
  • stop : This will be called only once and this can be used to release the acquired resources for processing.
  • currentState : Used to collect the serialize state of the processing element, that need to be persisted for the reconstructing the element to the same state on a different point of time.
  • restoreState : Used to restore serialized state of the processing element, for reconstructing the element to the same state as if was on a previous point of time.
After implementing this class, you need to write test cases in SampleWindowTestCase.java class. If you write your test cases you could test whether your code logic is correct or not by giving some dataset.

In Sample.siddhiext you have to give a name and the main class which is having the logic like this. SampleWindow=org.wso2.extension.siddhi.window.sample.SampleWindow

Then you have to build the Project using "mvn clean install" in terminal pointing this project's pom.xml. Then you will get jar file in target folder. Take that jar file to <CEP_HOME>/repository/components/dropins. After starting the CEP, you can use query like this 
"from <ImportEventName>#window.Sample:SampleWindow(ip,3)"

There is already I created a siddhi extension called UniqueLength. UniqueLength means according to the given attribute it will give unique events within given length. You can found the source code from here. After cloning, you can build and put it in jar file in <CEP_HOME>/repository/components/dropins and use this query to use this extension functionality.

/* Enter a unique ExecutionPlan */
@Plan:name('ExecutionPlan')

/* Enter a unique description for ExecutionPlan */
-- @Plan:description('ExecutionPlan')

/* define streams/tables and write queries here ... */

@Import('TestUniqueWindowIn:1.0.0')
define stream UniqueIN (timeStamp long, a string, ip string);

@Export('TestUniqueWindowOut:1.0.0')
define stream UniqueOUT (a string, ip string);

from UniqueIN#window.unique:length(ip,3)
select a, ip
insert expired events into UniqueOUT;


Here my siddhiext file name is unique.siddhiext and value in that file is length=org.wso2.extension.siddhi.window.uniquelength.UniqueLengthWindowProcessor


Hope you have got at least basic idea of Window extension. Have a nice day. :)

Tuesday, May 24, 2016

Streaming File send with File Inbound/File Connector

Hi all,

I think you are familiar with connector, streaming and VFS transfer. If not please refer my previous blogs. As I mentioned in the previous blog we can do the vfs sending feature by using file connector and file inbound. File connector is quite simple to use compared to the vfs sender for various file operations. Before that we will get some knowledge about inbound.

Brief intro to inbound endpoint

Inbound is a feature in WSO2 ESB. An inbound endpoint is a message entry point that can inject messages directly from the transport layer to the mediation layer, without going through the Axis engine. The following diagram illustrates the inbound endpoint architecture. For proxy, we have to invoke it. but a inbound can be invoked by itself. There are three types of inbounds in WSO2 ESB.
  • Listening Inbound Endpoint: A listening inbound endpoint listens on a given port for requests that are coming in. When a request is available it is injected to a given sequence.
  • Polling Inbound Endpoint: A polling inbound endpoint polls periodically for data and when data is available the data is injected to a given sequence.
  • Event based Inbound Endpoint: An event-based inbound endpoint polls only once to establish a connection with the remote server and then consumes events.
Here we mostly focus on Polling Inbound Endpoint as it deal with file in periodic manner. Polling inbound endpoint available with WSO2 ESB are given below.
  • File Inbound Protocol
  • JMS Inbound Protocol
  • Kafka Inbound Protocol
File Inbound Protocol 

The WSO2 ESB file inbound protocol is a multi-tenant capable alternative to WSO2 ESB VFS transport. In previous post we got the idea about VFS transport. Same functionality can be done through file inbound. we can specify streaming facility in WSO2 ESB 5.0 in file inbound. For Streaming we have to add message builder and formatter in the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
  • Add <messageFormatter contentType="application/file"
            class="org.wso2.carbon.relay.ExpandingMessageFormatter"/> under the message formatters.
  • Add  <messageBuilder contentType="application/file"
            class="org.apache.axis2.format.BinaryBuilder"/> under the message builders.

Steps 
  •  Start ESB 5.0 by going to <ESB_HOME>/bin.
  •  You can get file connector from the store (https://store.wso2.com/store/assets/esbconnector/aec1554a-29ea-4dbb-b8c5-5d529a853aa2).
  •  Add the connector by going to Connectors->Add in WSO2 ESB management console and browse connector zip file and enable it.
  • Go to Service bus-> sequence  in ESB management console and add following lines in source view to specify the connector configuration.

    <?xml version="1.0" encoding="UTF-8"?>
    <sequence xmlns="http://ws.apache.org/ns/synapse">
           <property name="OUT_ONLY" value="true"/>
           <property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
           <fileconnector.send>
            <address>file:///xxx/xxx/Desktop/testESB/OutTest</address>
            <append>true</append>
           </fileconnector.send>
        <drop/>
    </sequence>


    Here we specify the where do we want to send the file in address. Then save this sequence in registry.
  • Then go to Service bus->Inbound Endpoints-> Add Inbound Endpoint. After that specify the name and select Type as File. Then press next. In next window you have to specify sequence. u already saved your sequence in registry. Then you can select configuration registry. Then you will get following window and select your sequence.


 In error sequence you can put as fault. In suspend you initially put as false. If suspend is true, then the file inbound will not invoke. Put sequential and coordination as true. Specify the FileURI where you have the file and specify content type. if you want to use streaming specify contentType as application/file. Then click on show advanced options. In that you can specify the streaming as true if you want to send big files. ActionAfterProcess is the one what do you want to do after send the file to that specific location. you can specify MOVE or DELETE. we have to specify this to avoid recursive sending the same file again and again. You can get details of other parameters in here. Then save the inbound.
  • When you put a file in the fileURI location, you can see the response file in the address location you specified.
Note: You can use file send method in File connector not only for VFS transport, but also HTTP, JMS, FTP to send files.

Tuesday, May 17, 2016

Streaming VFS File Transfer using WSO2 ESB

Brief idea for VFS

VFS means Virtual File System. The purpose of a VFS is to allow client applications to access different types of concrete file systems in a uniform way. A VFS can, for example, be used to access local and network storage devices transparently without the client application noticing the difference.

Brief Intro to WSO2 ESB VFS transport with streaming

Through this we can pick up a file from a directory and process it within the ESB. You can get more about WSO2 ESB VFS transport from here. But when we using this VFS configurations for transferring large files (Greater that 500 MB), we get an Out Of Memory exceptions. In this time we can use streaming facility. To enable streaming, we have to add message builder "org.apache.axis2.format.BinaryBuilder" for this. Apart from that  we need to include the property "ClientApiNonBlocking" in the proxy configuration. To use vfs transport facility from WSO2 ESB, there are some steps you have to follow.
  1. Enable the VFS transport by editing the <ESB_HOME>/repository/conf/axis2/axis2.xml file and uncomment the VFS listener and the VFS sender as follows:
    <transportreceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>
    ...
    <transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>
  2.  Add message builder and formatter in the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
    • Add <messageFormatter contentType="application/file"
              class="org.wso2.carbon.relay.ExpandingMessageFormatter"/> under the message formatters.
    • Add  <messageBuilder contentType="application/file"
              class="org.apache.axis2.format.BinaryBuilder"/> under the message builders.
       
  3. You have to add a proxy like below.

    <?xml version="1.0" encoding="UTF-8"?>
    <proxy xmlns="http://ws.apache.org/ns/synapse"
           name="FileProxy"
           transports="vfs"
           startOnLoad="true"
           trace="disable">
       <description/>
       <target>
          <inSequence>
             <log level="custom">
                <property name="FileProxy" value="Processing file"/>
             </log>
             <property name="OUT_ONLY" value="true"/>
             <property name="ClientApiNonBlocking"
                       value="true"
                       scope="axis2"
                       action="remove"/>

             <send>
                <endpoint name="FileEpr">
                   <address uri="vfs:file:///home/Desktop/testESB/out"/>
                </endpoint>
             </send>

          </inSequence>
       </target>
       <parameter name="transport.vfs.Streaming">true</parameter>
       <parameter name="transport.PollInterval">20</parameter>   <parametername="transport.vfs.ActionAfterProcess">MOVE</parameter>
       <parameter name="transport.vfs.FileURI">file:///home/Desktop/testESB/in</parameter>
       <parameter name="transport.vfs.MoveAfterProcess">file:///home/Desktop/testESB/original</parameter>
       <parameter name="transport.vfs.MoveAfterFailure">file:////home/Desktop/testESB/failure</parameter>
       <parameter name="transport.vfs.Append">true</parameter>
       <parameter name="transport.vfs.Locking">enable</parameter>
       <parameter name="transport.vfs.FileNamePattern">.*.zip|.*.txt</parameter>
       <parameter name="transport.vfs.ContentType">application/file</parameter>
       <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
    </proxy>

    Here we specify ActionAfterProcess for doing some process after that file send to the specific location. We can specify MOVE/DELETE for this.

    We can specify source file to send from transport.vfs.FileURI. Destination of the file where we want to send that file can be specified in <address uri="vfs:file:///home/Desktop/testESB/out"/>

    This can be done through file inbound/ file connector. Next blog will cover about it. Until that bye. :)