Posted on

Kafka

Apache Kafka is an open source distributed streaming platform. It allows to publish/subscribe to data feeds (stream), it stores the data in a fault tolerant way and it allows consumers process the data as per the consumer’s need.

Key concepts in Kafka

It runs in a cluster, with configurable replication across the nodes.

Kafka
Kafka – source: https://kafka.apache.org/intro
  • Topics – These can be considered as message queues where publisher post their messages and consumers pick their messages.
  • Record – each record consists of a key, value and a timestamp.
  • Producer API – It allows publishers aka producers to post the messages or stream of messages to one or more topics in Kafka.
  • Consumer API – it allows consumers to poll the topics and fetch messages. Consumers can subscribe to one or more topics.
  • Stream API – it consumes the messages posted by publishers, processes them and posts the output messages back to Kafka system topics. Essentially they transform the messages.
  • Connector API – These connect Kafka topics to various applications and/or data sources.

History of Kafka

This platform was originally developed by the LinkedIn team and was eventually made open source. It is written in Scala and Java language. This platform can be used as a storage system, messaging system or simply stream processing. It achieves high throughput and low latency by batching the messages to optimize the n/w overhead and by using Java NIO. At its core, Kafka uses a simple binary protocol over TCP to communicate amongst various components.

Guarantees given by the platform:

  1. Records will always appear in the same sequence in the queue as they were received from the producers
  2. A consumer will see the records in the same sequence as they appear in the queue
  3. A topic with replication factor N, will be usable up to N-1 server failures, without losing any data.

Sample use cases

  • Message broker – can be compared with ActiveMQ or RabbitMQ
  • Website activity tracking – pushing various data points and then writing consumers to process the data
  • Log aggregation across various nodes

Related Links

Related Keywords

ActiveMQ, RabbitMQ, AMQP, Broker

Posted on

Material Design

Don’t get worried by reading the subject. This is not about any civil topic or interior design work. Material Design is a design concept introduced by Google which is prompting designers to rethink the web and mobile app UI. Here material is a metaphor. It is used to help you visualize web or mobile app as a physical object which has it’s own properties and responds to touch, zoom, pinch etc. Applications following material design present uniform behavior across various applications as well as various sizes of devices, thus giving uniform experience to the user.

Where can I see Material Design in action?

If you are on the android phone, it is everywhere in native apps (Android 5 onwards). When you use Chrome browser, check out settings and you will see it there.  If you click on a text field, the placeholder quickly moves to top left. If you click on a tab, the bottom border of the tab is highlighted. The buttons have a shadow effect giving a feeling of depth. All such things are based on Material Design principles.

Material Design

So what are the principles, this is based upon?

  • Each component occupies its own layer as if two physical objects are kept one on top of the other. They will not merge with each other, nor will the pass through each other. In simple terms – they appear as 3D objects.
  • Visual effect on one component will not follow through on other objects.
  • Some components would react to action (like touch, swipe, zoom etc) faster than others.
  • Every user interaction with the screen gives instant feedback to the user in terms action happened.
  • Components are aware of their surroundings. E.g. Upon touching a button, it pushes out certain components to make the place for the input field.
Material Design
Material Design – Example screen – Source: https://material.io

Similar to Bootstrap, there are several components pre-built and available for use to the developers. Refer to this article which has excellent comparison between Bootstrap and Material Design

In addition to Google’s own documentation, there are some very comprehensive articles about this design, check reference section below.

Related Links

Related Words

Bootstrap, Mobile App, Web App, Design, CSS

Posted on

GraphQL

GraphQL is a query language that was developed by Facebook for their internal development. However, later it was made open source in 2015. It is gaining traction since then but has a lot to cover to be called as popular.

GraphQL
GraphQL – query language to fetch and manipulate data

What are the main advantages of GraphQL?

GraphQL highly simplifies the fetching and manipulation of data, even when the operations tend to be complex. In “fetch” type of query, it provides exact data as requested by the client – no more no less. This makes the communication highly efficient. The important advantage is that the response format is specified by the client and is not decided by the server. Also, the structure of the data is not hardcoded as in REST APIs.

GraphQL makes the complex request very simple to implement as compared to REST APIs. e.g. additional filters and options both are pushed into query string parameters when using REST APIs. And that makes the request unreadable and difficult to understand which one is a filter and which one is an option. whereas GraphQL allows forming queries in a nested format, making it readable as well as easy to process.

GraphQL Server implementation is available in several languages including Javascript, Python, PHP, Ruby etc. Readymade server implementation from Facebook is also available, which uses Node.js.

Types of GraphQL operations

As can be guessed, it supports two types of operations:

  • Fetch data
  • Manipulate data (Create, Update, Delete)

Typically, you use GET request to fetch the data and POST request to manipulate the data. Using GET allows the responses to be cached at various levels and adds to the efficiency of the overall architecture.

Both REST and GraphQL support evolution of APIs i.e. support for additional fields and deprecation of unused fields, without impacting any of the endpoints.

You can read more about REST and GraphQL here and here. But essentially, both have their own advantages and in a microservices architecture, both could live together happily!!

Related Links

Related Keywords

Microservices Architecture, REST, Node.js

Posted on

Serverless Computing

Serverless Computing is a computing model where servers are not used. Are you kidding me? You are right. This is not a case. Serverless Computing is actually a misnomer. What it really means is developers are freed from managing the servers and are free to focus on their code/application to be executed in the cloud.

Serverless Computing
Serverless Computing – You won’t need such a huge space!!

So, how does Serverless Computing really work?

In this computational model, you only need to specify minimum system requirements such as RAM. Based on that the cloud provider would provision the required resources such as CPU and network bandwidth. Whenever you need to execute your code, you hit the endpoint and your code is executed. In the background, the cloud provider, provisions the required resources, executes the code and releases the allocated resources. This allows cloud provider to use the infrastructure for other customers when they are idle. This also gives out the benefit to the customers as they are not charged for the idle time. Win-Win situation for both, isn’t it?

Serverless Computing is popular in microservices architecture. The small pieces of code which form the microservices can be executed using Serverless components very easily. On top of it, cloud provider also handles the scaling up. So if there are more customers hitting your serverless endpoint, cloud provider would automatically allocate more resources and get the code executed.

Any precautions to be taken?

Absolutely. Every developer needs to keep few things in mind while developing code for Serverless computing:

  • The code can’t have too many initiations steps, else it will add to the latency.
  • The code is executed in parallel. So there shouldn’t be any interdependency which would leave the application or data in the incoherent state.
  • You don’t have access to the instance. So you can’t really assume anything about the hardware.
  • You don’t have access to local storage. You need to store all your data in some shared location or central cache.

So who all provide this option?

  • AWS (obviously!!) was the first one to provide this (2014) – AWS Lambda.
  • Microsoft Azure provides Azure Functions
  • Google Cloud provides Google Cloud Functions
  • IBM has OpenWhisk which is open source serverless platform

Support for languages varies from provider to provider. However, all of these offerings support Node.js. Among other languages, Python, Java are more popular.

Related Links

Related Keywords

Serverless Architecture, API Gateway, AWS S3, Microservices Architecture

Posted on

Node.js

Let’s know about a leader in event-driven architecture – Node.js. This is an open source and cross-platform runtime environment built using JavaScript for server-side script execution. Javascript was historically used for script execution on the client side (i.e. browser). However, this environment allows developers to use the same language on both sides that server side as well as client side.

Why is Node.js so popular?

Node.js gained popularity because of the ease of doing simple tasks and the scale that this environment can provide. It implements the event-driven architecture and async I/O. That means, whenever a client requests data, an event is added to the event queue. An Event loop constantly picks up the events in the queue. If any request can be responded to, without any blocking operations such as reading data from file system or database, it is responded immediately. If the operation is time-consuming as in case of DB operation, the event is handed off to the internally maintained thread pool and a call back is registered for this event. While the thread is executing the code in the backend, the main thread is free to process next request. This increases the throughput of Node.js server tremendously. When the DB operation is completed, the response is passed on back to the client through the callback mechanism.

Node.js model
Node.js representation. Source: https://strongloop.com/strongblog/node-js-is-faster-than-java/

Since there are already quite a few articles available on this subject, I am going to list them here for your ready reference:

 

What are the common use cases for Node.js?

Whenever you are intending to have a constant connection between server and client, Node js becomes one of the candidates. Typical examples are – a chat server, online game, Streaming etc.

Some trivia about Node.js

  • Created by Ryan Dahl in 2009
  • Uses V8 Javascript engine by Google (which is also used in Google Chrome)
  • Cross-platform – runs on Linux as well as Windows.
  • Outperforms Apache + PHP and rivals the performance with Nginx.

Related Links:

Related Keywords:

Ruby On Rails, Django, Apache, Nginx, Event Driven Architecture (EDA), Software Architecture, Non-blocking I/O, Async I/O