Introduction to Moses

Before coming to the Moses we need to know brief introduction to Natural Language Processing and Language translation. Then we could understand the Moses easily.

Natural Language Processing

Natural Language Processing is an Artificial Intelligence method which is used to communicate with Intelligent system such as Computers using natural language such as English, Tamil and Sinhala. By utilizing NLP, developers can organize and structure knowledge to perform tasks such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation. NLP considers the hierarchical structure of language: several words make a phrase, several phrases make a sentence. NLP is commonly used for text mining, machine translation, and automated question answering.

We can use NLP to translate from one language to another language. Instead of hand-coding large sets of rules, NLP can rely on machine learning to automatically learn these rules by analyzing a set of examples (i.e. a large corpus) and making a statistical inference.

Moses

Moses is a statistical machine translation system which allows you to translate from one language to another language by training translation models. For training the model you need collection of translated texts in both language (parallel corpus). Once you have a trained model, an efficient search algorithm quickly finds the highest probability translation among the exponential number of choices. It is a data driven machine translation approach. Moses system based on the Bayes theorem. 

If we explain this in translation point of view, probability of translation from f language to e language is depend on probability of translation from e language to f language and probability of  e language.

Further the system can be drilled down as log linear models.

• Weight-t Translation
• Weight-l Language model
• Weight-d distortion (reordering)
• Weight-w word penalty

Moses was developed in C++ for efficiency and followed modular, object-oriented design.The toolkit is a complete out-of-the-box translation system for academic research. It consists of all the components needed to preprocess data, train the language models and the translation models. It also contains tools for tuning these models using minimum error rate training and evaluating the resulting translations using the BLEU score.

 

 Moses requires two main things

• Parallel text- Collection of sentences in two different languages, which is sentence-aligned, each sentence in one language is matched with its corresponding translated sentence in the other language.
• Monolingual target set- A statistical model built using monolingual data in the target language and used by the decoder to try to ensure the fluency of the output.

There are two main components in the Moses

• Training Pipeline- Take the raw data and turn it into a machine translation model
• Decoder- Translate the source sentence into the target language

Decoder Modules 

• Input: This can be a plain sentence, or it can be annotated with xml-like elements to guide the translation process, or it can be a more complex structure like a lattice or confusion network.
• Translation model: This can use phrase-phrase rules, or hierarchical (perhaps syntactic) rules.
• Decoding algorithm: Decoding is a huge search problem.
• Language model: Moses supports several different language model toolkits (SRILM, KenLM, IRSTLM, RandLM)

 The process of Moses system is described in a picture is given below.



 We need to install Moses to train the system and get output. So in the next blog we will see how to install Moses in the Ubuntu operating system.

Enterprise Application Integration and Enterprise Serivce Bus

Enterprise Application Integration

• Integrating systems and applications together
• Get software systems to work in perfect synchronism with each other
• Not limited to integration within an organization
• Integrating with customer applications
• Integrating with supplier/partner applications
• Integrating with public services

By using EAI, we have got another problem that how can we talk to each different services which develop on different technologies, different languages, different protocols, different platforms, different message formats and different QoS requirements (security, reliability). ESB is the rescue for this problem. 

Now we will see how can we use ESB to resolve this problem. Think a real scenario. A Chinese girl is joining who does not know English in your classroom. Think you know only Englis and you don't know Chinese. So how can you communicate with that Chinese girl? In this scenario, you can a friend who knows Chinese and English. Through that, you can easily communicate with that girl. This is cost and time effective as you don't need to study Chinese. 

Now we can apply this solution to the software industry. Let's assume, you are developing a business application for buying a dress. There you need to talk to Sales System, Customer System, and Inventory system. In this example, let's assume sales system built using SOAP protocol (Exposing SOAP services). Customer system using XML based REST services and Inventory system using JSON-based REST services. Now you need to deal with multiple protocols. Here we can use ESB as the rescue. 

What is ESB?

The normal bus is used to transfer from one place to another place. In ESB, you need to pass your message to ESB, ESB will pass your message to a destination. Also if destination sends a response, ESB will take that response and deliver to you. In the previous example, sales system will send the soap message to ESB. ESB will take this message and convert it to XML based REST message to the cusomer system. You may connect to multiple application through ESB. But you only need to do one simple connection which calls ESB only. ESB will talk to rest of the applications.

Introduction to SOA (Service Oriented Architecture)

Hi all,

In this blog, we will see about What is SOA in a simple way with real world examples.

Before coming to the point what is SOA, we need to know why SOA is needed and why it had been evolved. For this, we will go with simple real world example. Think about an old radio. In there everything is integrated such as FM radio, The cassette player, the speaker ... But if we want a double cassette player or CD player, we have to change the whole thing again and again. But with modular applications, each part is independent. We can add other items to the already available thing. There should be a way to communicate with each component.

Let's apply this scenario to the software industry. Initially, we used standalone application which is run on one computer and do one job. Database, UI, everything is in the simple computer. Then there was a requirement that multiple users need to access at the same time. For that, we got Client Server architecture. This means you have the front end on your machine and database logic & rest of the things in the different machine which is called as a server. Every client calls the same server machine. Then requirements had been grown. So people moved to different architecture 'Multi-tier architecture'. The front end is on your machine. Business logic implemented in different server and DB is on another server. After that people decided to go in distributed applications. For example, one application does a part of the job, another application does that's job and the third application does another job. By integrating all these jobs, you can fulfill your requirement. That means different services and different responsibilities owned by the different applications.

Here you have other problem that how can we inter-connect with these applications. Practically application A can run on the Linux which is implemented using JAVA, Application B runs on windows which are implemented in C#. Here JAVA application needs to communicate with the C# application. So we need the new model. To overcome this scenario we came u with SOA model. SOA means Service Oriented Architecture.

So we need to know what is service? When we connect to the application, the application may not expose everything that it can do. But it may expose certain functionalities to the world. For example, Hotel reservation system may expose register, login, get booking details and book rooms. But all other private functions will keep privately. When they expose the functionalities,  we called it as service. We depend on multiple services to achieve a specific goal. This is what service oriented architecture.

"A set of principles and practices for modeling enterprise business functions as services or micro services which have following attributes."

The features of SOA are

• Standardized: Support open standards.
• Loose Coupling: Need to give required data to the interface and expects the response. Processing will be handling by the service.
• Stateless: The Service does not maintain state between invocation
• Reusable
• Autonomic
• Abstract
• Discoverable

Couples of example for SOA

• The supply chain management system should keep an eye on the inventory management system
• Decision support systems cannot help make useful decisions without insight to all the aspects of the business

In next blog, we will see some more about SOA and WSO2 ESB.

Working with cloud instances in WSO2

Hi all,

I had a chance to work with cloud instances in WSO2 environment. There are four instances which are created in ubuntu environment. I have had cluster set up of four nodes in cloud.

In this blog I am not going to explain the cluster setup. Here there will be a brief explanation of cloud instances and how can we handle it.

When we are creating cloud instance we will get a key for that. After that we need to change mode of the file by following command.

    change key file permission : chmod 600 <file>

Then we can  start the instance by specifying the IP address of the node. 

    ssh -i <key file> ubuntu@<IP>

When we starting it, it will ask passpharase for key. we need to give that value.

Now we have started the cloud instance.

The cloud instance will look like pure computer when we buy it (Except OS might be installed). So we need to all other things from the terminal.

• Command to download from a web link.
  
  sudo apt wget <link>
  
• To install unzip
  
  sudo apt get unzip
  
• Need to install JAVA as same as installing JAVA in ubuntu [1]
  
• If you want to copy from local machine to cloud instance, you can use sftp.
  
  Need to start sftp in the cloud instance.
  
            sftp -i <key file> ubuntu@<IP>
  
  Copy file
  
            put <FROM> <TO>

References

 [1] https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get

RabbitMQ SSL connection Using RabbitMQ Event Adpter in WSO2 CEP- Part 2

stHi all,


This is the continuous blog of this blog. In the previous blog we saw about rabbitMQ broker side configuration. Here we are going to see about specifying CEP side Configuration. RabbitMQ Receiver is an extension for the CEP. So we need to download it and put it into the CEP. Those are given below.

Steps 

1.  Please visit this link to download RabbitMQ Receiver extension.
2. Download rabbitmq client jar from this link. 
3. Place the extension .jar file into the <CEP_HOME>/repository/components/dropins directory which you downloaded in step 1.
4. Place the rabbitmq client .jar file into the <CEP_HOME>/repository/components/lib directory which you downloaded in step 2.
5. Start the CEP Server by going to directory <CEP_HOME>/bin in terminal and giving command like sh ./wso2server.sh
6. Create a Stream in CEP. Refer more here.
7. Go to Main => Manage => Receiver to open the Available Receivers page and then click Add Event Receiver in the management console. You can get the management console by visiting the https://172.17.0.1:9443/carbon
8. Select Input Event Adapter Type as rabbitmq.
9. Then you will get a page like below.

   9. Then you need to specify values for the properties given above. Here I have mentioned important and  mandatory properties' values.

• Host Name: localhost
• Host Port: 5671
• Username: guest
• Password: guest
• Queue Name: testsslqueue
• Exchange Name: testsslexchange
• SSL Enabled: True
• SSL Keystore Location: ../client/keycert.p12
• SSL Keystore Type: PKCS12
• SSL Keystore Password: MySecretPassword
• SSL Truststore Location: ../rabbitstore (These are the values we took from the part 1 blog step 5)
• SSL Truststore Type: JKS
• SSL Truststore Password: rabbitstore (The password is the one we used when we are key tool command in Step 5)
• Connection SSL Version: SSL

After specifying above values, select the stream you had add in step 3 and save it. 

Then if there is a message in the queue you mentioned in RabbitMQ broker, you will get it into the CEP. The data from the queue can be used in further in Execution plan for processing and we can publish it.