Siddhi is a 100% open source Complex Event Processor(CEP) offered by WSO2, Inc. It was initially started as a research project at University of Moratuwa, Sri Lanka. The main application of Siddhi is processing data streams in real time. It is written in Java and thoroughly optimized for high performance.
Siddhi is used by many companies including Uber, and eBay (via Apache Eagle). In fact Uber processes more than 20 billion events per day using Siddhi. Before diving into using Siddhi we will first discuss about CEP.
(Full disclosure: I was an intern at WSO2 from July 2017 to December 2017 during which I wrote this blog)
This blog has 5 sections as follows,
- Overview on Stream Processing and Complex Event Processing — about the domain of Siddhi
- Overview of Siddhi — basic architecture explained
- Using Siddhi for the First Time — how to set up the software
- Siddhi ‘Hello World!’ — Your First Siddhi Application
- Simulating Events — testing your query with simulated events
1. Complex Event Processing (CEP)
First we will look at what an event is through an example. If we take the transactions through an ATM as a data stream, one withdrawal from it would be an event. This event will contain data about amount, time, account number etc. Many such transactions will make up a stream.
Gartner’s IT Glossary defines CEP in the following way,
CEP is a kind of computing in which incoming data about events is distilled into more useful, higher level “complex” event data that provides insight into what is happening.
CEP is event-driven because the computation is triggered by the receipt of event data. CEP is used for highly demanding, continuous-intelligence applications that enhance situation awareness and support real-time decisions.
Basically a CEP receives data event-by-event and processes them in real time to give meaningful information.
2. Overview of Siddhi
As you can see above Siddhi,
- can accept event inputs from many different types of sources
- process it to generate insights
- publish them to many types of sinks.
To use Siddhi a user will write a Siddhi query which will be discussed in the 4th section. After writing a Siddhi query and starting a Siddhi application, it will
- take data event-by-event
- process the input data in each event
- add the high level data generated to each event
- send them to the output stream.
3. Using Siddhi for the First Time
We will use WSO2 Stream Processor(will be addressed as ‘SP’ hereafter) — a server version of Siddhi that has sophisticated editor with GUI (called “Stream Processor Studio”)where you can write your query and simulate events in a data stream.
Step 1 — Install Oracle Java SE Development Kit (JDK) version 1.8
Step 2 — Set the JAVA_HOME environment variable
Step 3 — Download the WSO2 Stream Processor
Step 4 — Extract the downloaded zip and navigate to <SP_HOME>/bin (SP_HOME refers to the extracted folder)
Step 5 — Issue the following command in command prompt(Windows)/terminal(Linux)
For Windows: editor.bat
For Linux: sh editor.shAfter successfully starting the SP the terminal should look like this in Linux,
Refer to the SP Quick Start Guide for more information. After starting the WSO2 Stream Processor access the Text Editor by visiting the following link in your browser
http://localhost:9090/editorAfter successfully starting the SP and accessing the editor it should display the following,
4. Siddhi ‘Hello World!’ — Your First Siddhi Query
Siddhi QL is a rich, compact, easy-to-learn SQL-like language. We will first learn how to add data from events one-by-one and output total value with each event. Siddhi has lot of in-built functions and extensions available for complex analysis but we will start with a simple one. You can find more information about the grammar and the functions in Siddhi Query Guide.
We will consider a scenario where we are loading cargo boxes into a ship. We need to keep track of the total weight of the cargo added. Measuring the weight of a cargo box when loading is treated as an event. We can write a Siddhi program for the above scenario which will have 3 parts.
Part 1 — Giving our Siddhi application a suitable name. This is a Siddhi routine. Here we will name our app as “HelloWorlApp”
@App:name("HelloWorldApp")Part 2 — Defining the input stream. This will have a name for the input stream, name and type of the data coming with each event. Here,
- Name of the input stream — “CargoStream”
- name of the data in each event — “weight”
- Type of the data “weight” — int
define stream CargoStream (weight int);Part 3 — The actual Siddhi query. Here we will specify 4 things.
- A name for the query — “HelloWorldQuery”
- which stream should be taken into processing — “CargoStream”
- what data we require in the output stream — “weight”, “totalWeight”
- which stream should be populated with the output — “OutputStream”
@info(name='HelloWorldQuery')
from CargoStream
select weight, sum(weight) as totalWeight
insert into OutputStream;
You can see a different query from lines 10–12 in the above picture. This is to see the events from “OutputStream” in the terminal we used to start SP Text Editor. Just copy and paste it in your code.
from OutputStream#log("LOGGER")
select *
insert into temp2;5. Simulating Events
The Stream Processor Studio has in-built support to simulate events. You can do it using the “Event Simulator” panel in the left side of the Stream Processor Studio. You should save your HelloWorldApp by going to file -> save before event simulation. Then click on the “Event Simulator” and configure it as shown below.
Step 1 — Configurations,
- Siddhi App Name — “HelloWorldApp”
- Stream Name — “CargoStream”
- Timestamp — (Leave it blank)
- weight — 2 (or some integer)
Step 2 — Click on “Run” mode and hit the “Start” button to start the Siddhi App. This will print a message “HelloWorldApp.siddhi Started Successfully!” in the Stream Processor Studio console.
Step 3 — Click on the “Send” button and observe the terminal where you started WSO2 Stream Processor Text Editor. You could see a log that contains “outputData=[2, 2]”. Click on send again and you can see a log with “outputData=[2, 4]”. You can even change the value of the weight and send to see it getting summed up.
Bravo! You have successfully finished you Siddhi Hello World! You can learn many other complex functions by referring to the Siddhi Query Guide.
Content of this blog was extracted to prepare Quick Start Guide for Siddhi at https://wso2.github.io/siddhi/documentation/siddhi-quckstart-4.0/ to which I have contributed
