Serverless Eventing: A Series

“I want a data-driven organization!” We have all heard this in recent years haven’t we? Maybe it was at an All-Hands, maybe your manager, probably at a mixer in Silicon Valley somewhere. My healthcare provider told me about their new “data-driven matching”.

Sometimes I wonder if the term holds any weight anymore or if it has just become a new marketing term that we use to sound futuristic. Are they collecting data? What data are they collecting? How are they extracting meaning from the data? Is it being applied properly? These are all very good questions when evaluating what it means to be “data-driven”.

Depending on who you ask, you will get a lot of answers. I will break down my opinion on the concept.

Let’s first break down the concept of data. We don’t need to go too deep as data is information. Your weight, your birthday, your name, and more. I tend to see this as static data. This data largely persists through time and changes are infrequent. This data has value to many organizations. I may want to send a “Happy Birthday” email to my customers when it’s their birthday or offer them 10% off of a service. Knowing where a customer resides allows you to market specials at the closest location of your grocery chain.

Data in and of itself is useless though. I can collect every iota of data in the world but if it sits in a storage bucket somewhere, what use is it? Data should be collected in order to be analyzed and derive meaning. Why did people purchase more pizza in April than in February? What are the most clicked parts of your website during the summer months?

When you are able to analyze and derive meaning, you then need to apply that meaning towards decision making? Maybe books on baking aren’t as popular in the spring months are they are the autumn months so you should consider that when stocking your bookstore. THAT is what I think about when I hear data-driven. It means collecting, analyzing, and using data to drive the decisions in you or your organizations lifecycle.

In the age of IoT, Smart Phones, and Responsive websites, however, we have created a new breed of data. According to IDC, nearly 30% of all data will be real-time in nature. This data is dynamic in nature. If I am using a fitness app to track my running, every second I am in a new location. The value of a given crypto can change every second as well.

This dynamic data I often frame as events. This is more than just information, this is information that is time-bound. If I am waiting for a ride share, I want to know if they are 5 minutes away or 15. I want to know what traffic looks like right now as I head home from work, not a week ago.

This new advent of real-time dynamic data has pushed us from needing to be just data-driven to being event-driven. If we are receiving data in real-time that is time-critical, we need to analyze and act on it just as fast.

This brings us to SERVERLESS EVENTING. Now what is “Serverless Eventing”? This isn’t trying to send event data directly through the sky and bypassing the need for bare-metal. Serverless is about simplifying the developer experience and enabling them to manage their own deployments. This allows developers to focus more on code and packaging code and less on managing networks, routes, storage, server uptimes, and everything else infrastructure related.

Serverless Eventing is simplifying the developer experience in building real-time streaming applications. Streaming applications usually use a message queue or bus and then you hardcode the functionality imperatively. You are writing libraries and functions to connect to your Kafka topics and repeating this for every service you create.

Serverless Eventing allows you to declaratively define your streaming pipeline. It offers abstractions to simplify connecting code to events. These abstractions are also reusable and scalable. As your application grows to have more services, you can declaratively connect your events to your applications.

I am starting this series to cover some concepts and examples related to Serverless Eventing. This will center around Knative Eventing. Knative is a Kubernetes Native platform that was open sourced in 2018 by Google. It enables operators to design serverless platforms for their developers.The Eventing component focuses on allowing developers to declaratively build serverless applications.

I will show a variety tutorials and concepts using Knative Eventing stand-alone as well as showing how to integrate it with legacy systems such as Apache Kafka. You can follow my code and demos at this GitHub Repo.

Please feel free to share and ping me with questions.