A Beginner’s Guide to Siddhi — Complex Event Processor

Siddhi is a 100% open source Complex Event Processor(CEP) offered by WSO2, Inc. It was initially started as a research project at University of Moratuwa, Sri Lanka. The main application of Siddhi is processing data streams in real time. It is written in Java and thoroughly optimized for high performance.

Source: http://www.bbc.com/news/business-37802386

Siddhi is used by many companies including Uber, and eBay (via Apache Eagle). In fact Uber processes more than 20 billion events per day using Siddhi. Before diving into using Siddhi we will first discuss about CEP.

()

This blog has 5 sections as follows,

  1. Overview on Stream Processing and Complex Event Processing — about the domain of Siddhi
  2. Overview of Siddhi — basic architecture explained
  3. Using Siddhi for the First Time — how to set up the software
  4. Siddhi ‘Hello World!’ — Your First Siddhi Application
  5. Simulating Events — testing your query with simulated events

1. Complex Event Processing (CEP)

First we will look at what an event is through an example. If we take the transactions through an ATM as a data stream, one withdrawal from it would be an event. This event will contain data about amount, time, account number etc. Many such transactions will make up a stream.

Individual Transaction Events Making up a Stream

Gartner’s IT Glossary defines CEP in the following way,

Basic Operation of Siddhi

Basically a CEP receives data event-by-event and processes them in real time to give meaningful information.

2. Overview of Siddhi

As you can see above Siddhi,

  • can accept event inputs from many different types of sources
  • process it to generate insights
  • publish them to many types of sinks.

To use Siddhi a user will write a Siddhi query which will be discussed in the 4th section. After writing a Siddhi query and starting a Siddhi application, it will

  1. take data event-by-event
  2. process the input data in each event
  3. add the high level data generated to each event
  4. send them to the output stream.

3. Using Siddhi for the First Time

We will use WSO2 Stream Processor(will be addressed as ‘’ hereafter) — a server version of Siddhi that has sophisticated editor with GUI (called )where you can write your query and simulate events in a data stream.

Step 1 — Install Oracle Java SE Development Kit (JDK) version 1.8

Step 2 — Set the JAVA_HOME environment variable

Step 3 — Download the WSO2 Stream Processor

Step 4 — Extract the downloaded zip and navigate to (SP_HOME refers to the extracted folder)

Step 5 — Issue the following command in command prompt(Windows)/terminal(Linux)

For Windows: editor.bat
For Linux: sh editor.sh

After successfully starting the SP the terminal should look like this in Linux,

Terminal after starting WSO2 Stream Processor Text Editor

Refer to the SP Quick Start Guide for more information. After starting the WSO2 Stream Processor access the Text Editor by visiting the following link in your browser

http://localhost:9090/editor

After successfully starting the SP and accessing the editor it should display the following,

Stream Processor Studio

4. Siddhi ‘Hello World!’ — Your First Siddhi Query

Siddhi QL is a rich, compact, easy-to-learn SQL-like language. We will first learn how to add data from events one-by-one and output total value with each event. Siddhi has lot of in-built functions and extensions available for complex analysis but we will start with a simple one. You can find more information about the grammar and the functions in Siddhi Query Guide.

We will consider a scenario where we are loading cargo boxes into a ship. We need to keep track of the total weight of the cargo added. Measuring the weight of a cargo box when loading is treated as an event. We can write a Siddhi program for the above scenario which will have 3 parts.

Source: https://ak3.picdn.net/shutterstock/videos/14807563/thumb/1.jpg?i10c=img.resize(height:160)

Part 1 — Giving our Siddhi application a suitable name. This is a Siddhi routine. Here we will name our app as “HelloWorlApp”

@App:name("HelloWorldApp")

Part 2 — Defining the input stream. This will have a name for the input stream, name and type of the data coming with each event. Here,

  • Name of the input stream —
  • name of the data in each event —
  • Type of the data
define stream CargoStream (weight int);

Part 3 — The actual Siddhi query. Here we will specify 4 things.

  1. A name for the query —
  2. which stream should be taken into processing —
  3. what data we require in the output stream —
  4. which stream should be populated with the output —
@info(name='HelloWorldQuery')
from CargoStream
select weight, sum(weight) as totalWeight
insert into OutputStream;
Hello World in Stream Processor Studio

You can see a different query from lines 10–12 in the above picture. This is to see the events from “OutputStream” in the terminal we used to start SP Text Editor. Just copy and paste it in your code.

from OutputStream#log("LOGGER")
select *
insert into temp2;

5. Simulating Events

The Stream Processor Studio has in-built support to simulate events. You can do it using the panel in the left side of the Stream Processor Studio. You should save your HelloWorldApp by going to file -> save before event simulation. Then click on the and configure it as shown below.

Step 1 — Configurations,

  • Siddhi App Name —
  • Stream Name —
  • Timestamp — (Leave it blank)
  • weight — 2 (or some integer)

Step 2 — Click on “Run” mode and hit the “Start” button to start the Siddhi App. This will print a message in the Stream Processor Studio console.

Step 3 — Click on the “Send” button and observe the terminal where you started WSO2 Stream Processor Text Editor. You could see a log that contains . Click on send again and you can see a log with You can even change the value of the weight and send to see it getting summed up.

Terminal after sending 2 twice

Bravo! You have successfully finished you Siddhi Hello World! You can learn many other complex functions by referring to the Siddhi Query Guide.

Content of this blog was extracted to prepare Quick Start Guide for Siddhi at https://wso2.github.io/siddhi/documentation/siddhi-quckstart-4.0/ to which I have contributed

Senior Software Engineer @ Sysco LABS, Former SE @ WSO2 Inc., ENTC grad @ University of Moratuwa, Sri Lanka. www.niruhan.com