Posted on

Munin

You are a system administrator, managing a set of servers. As a part of your job, you need to ensure that all the servers are up and running 24X7, they are resources are utilized optimally. And that no single server is overloaded or consuming huge memory or some similar abnormal behavior over a time period. Obviously, you will be needing a monitoring tool. Munin is one such tool, which helps system administrators to keep track of system resources.

Munin Architecture

Munin derives its name from mythology. It means memory. This monitoring software memorizes what it sees and hence the name.

This monitoring tool uses Master-node architecture. Master periodically pulls data from all the configured nodes and processes it to create graphical dashboards.

Munin Architecture
Munin Architecture – Source: http://guide.munin-monitoring.org/en/latest/architecture/index.html

One of the advantages of Munin is its vast plugin eco-system. There are several hundred plugins available, which make it a powerful monitoring system. Munin itself uses RRDTool, which is written in Perl, whereas plugins could be written in any language. This tool is available on various distros such as Debian, Ubuntu, and CentOS.

Munin-memory-week
Munin graph depicting memory usage over a week
Once data is collected from nodes, it is processed by the master and various graphs are made available on the dashboard. One of the components is – Munin-limits – which monitors the values between configured “ok”, “warn” and “crit” levels and can create a notification for the administrators.

Alternate Server Monitoring Tools

Several alternatives are available.

  • Nagios
  • NewRelic
  • Netdata
  • Zabbix
  • …

Considering that most of the deployments are in the cloud these days, you may want to use the monitoring tools provided by your cloud providers:

  • AWS Cloudwatch
  • Azure Monitor
  • Google Cloud – Stackdriver monitoring

Related Links

Related Keywords

Server Monitoring, AWS Cloudwatch, Azure Monitor, Nagios, NewRelic

 

Posted on

Filestack

You are launching a service, where your users need to upload their files to your app or web app. Your userbase is global. You are currently supporting only a handful of sources to pick files from. Also, it is observed that your users are leaving because file upload takes too long or fails due to a variety of issues. Filestack comes to your rescue.

But what is Filestack?

Filestack is a developer service that allows app developers to build apps which can connect to several sources and also upload the files reliably. It solves 2 problems:

  • Making uploads reliable and faster
  • Enabling users to upload files from several sources (25+)

The Filestack team has used an interesting model to make the uploads faster. It is a two-step process. The process uses an S3 bucket as an intermediate hop. S3 is a file storage solution by AWS, which has a presence across the globe. Using this network, Filestack uploads the user-provided file to the nearest S3 bucket. This turns out to be faster due to the elimination of network latency. As soon as the upload is completed, it returns URL to the application and application can start processing the file as per the need. In the background, the file from Filestack S3 bucket is moved to cloud storage specified by the app. The URL is updated to point to new location and app can continue to use that URL for rest of the lifecycle. So, even if it includes additional hop, it turns out to be faster from a user perspective.

Filestack - Content ingestion network illustration
Filestack – Content ingestion network illustration (Source: https://www.filestack.com/features/content-ingestion-network)

Filestack not only solves above mentioned two problems, but it also provides several additional functionalities for image files.

  • Image transformation such as adding borders, flipping the image, applying filters, image rotation
  • Resizing of images – on the fly
  • Image cropping
  • Collage
    You can check out the live examples here.

It also provides some advanced features based on ML (Machine Learning)

  • Object recognition
  • Object detection
  • OCR (Optical Character Recognition)
  • Copyright detection

Alternatives to Filestack:

Obviously, you can build your own connectors for various sources, but probably you would end up spending more time to implement all the functionality that Filestack is already providing. There have been some alternatives available:

  • UploadCare
  • Kloudless
  • Filespin.io

Related Links

Related Keywords

Machine Learning, Google Vision, AWS S3

Posted on

Vagrant

You are part of a development team and you want to try out the latest version of the VM that your application uses. Or you want to make some configuration changes and test them out in your development environment. However, you are not sure if your changes would be useful or successful. And hence you want an ability to roll back the changes or potentially version control them. How do you do it? Vagrant comes to the rescue!!

What is Vagrant?

Vagrant is an open source tool for building and maintaining portable development environments. With the rise of complex architectures involving several different servers and technology stacks, Vagrant simplifies the task of creating and maintaining the required stack of software/libraries etc.

Vagrant stores the configuration in the form of a text file(s) and these files could be put under your favorite version control system such as git. If changes don’t work out, you can easily roll back and go to the earlier working stage. This capability improves the development productivity a lot and hence Vagrant has become a darling of several development teams.

This tool uses a concept of “provisioners” and “providers”. Provisioners are tools that allow to customize/modify the environments – examples are Chef and/or Puppet. Whereas Providers are services which provide virtual machines such as AWS, Docker, VMWare etc.

Vagrant Workflow
Vagrant – workflow [Source: https://objectcomputing.com/resources/publications/sett/march-2015-docker-vs-vagrant]
Vagrant abstracts the machine. It sits as a wrapper on top of underlying hardware. You can throw away the vagrant configuration and create new one very easily, by simply changing the config files. It provides command line interface, which can work in any environment irrespective of the underlying VM or OS. This additional layer provides the portability, simplicity, and interoperability. One developer can easily share this environment she has created with others and other can quickly set up the replica of the shared environment.

One might be tempted to compare Vagrant with other configuration management tools such as Chef/Puppet/Ansible. However, Vagrant is commonly used along with one of these tools as they serve different purposes.

Related Links

Related Keywords

Configuration Mangement, Docker, Chef, Puppet, Ansible, VMWare, AWS, DevOps

 

Posted on

Paravirtualization

Paravirtualization is a type of virtualization, which allows us to install and run one or more different operating systems (Guest OS) on a single physical server (Host OS). You might get tempted to compare it with Hypervisor and you wouldn’t be wrong. They have similarities and differences.

A Hypervisor is a software that allows installing one or more different operating systems on a single physical server. The OS which runs the Hypervisor is called as Host OS and the OS which is run by Hypervisor is called Guest OS. Here, guest OS doesn’t have a clue that it is running in a virtualized environment. This is where Hypervisor differs from Paravirtualization.

So how does Paravirtualization work?

Paravirtualization
Paravirtualization – Source: https://wiki.xen.org/wiki/Paravirtualization_(PV)

In virtualization, Hypervisor acts as a relay between guest OS and host OS. In Paravirtualization, guest OS needs to know that it is interacting with Hypervisor and hence needs to make different API calls. This calls for modifications to the guest OS and hence creates a tight coupling between guest OS and Hypervisor used. Since proprietary OS providers such as MS do not allow OS level modifications and hence you would end up using open source operating systems such as Linux. A good combination is Linux with Xen server

You would be wondering if there are any real advantages of paravirtualization. Paravirtualization has shown performance improvements for certain applications or in certain use cases, but it has not shown the reliability. Especially with the rise of software level virtualization, which shows very high reliability, need for Paravirtualization has gone down. The cost associated with guest OS modifications, tight coupling with Hypervisor, is not getting offset by the not-so-predictable performance gains. So, you wouldn’t find many instances where paravirtualization is used.

Related Links

Related Keywords

Hypervisor, Virtualization, Containers

Posted on

Kubernetes

Kubernetes is a tool or a system for automating the deployment of containerized applications. In a loose analogy, one can say what Chef is for infrastructure deployments, Kubernetes is for container deployments.

History of Kubernetes

Kubernetes Logo
Kubernetes Logo – Source: GitHub

It was part of Google’s internal tool system called “Borg” which was later made as Open-source in 2015. Internal codename of this project was “Seven” and is reflected in the logo of this system.

Kubernetes has been maintained by Cloud Native Computing Foundation which is a collaboration between Google and Linux Foundation.

What does Kubernete offer?

It offers to manage the “immutable infrastructure” in an automated fashion. Typically, containers are launched using images which do not change. If you want to make changes to the software piece which is part of the container, you simply update the image, launch new containers and throw away the older containers. Kubernetes allows you to do this efficiently.

The lowest level abstraction that Kubernete provides is called a “Pod”. It consists of one or more containers which can run on a same “node”. A Node is a server or VM on which Kubernetes cluster is running. The containers in a Pod could be sharing resources and can communicate with each other. This allows running tightly knit services to be run from the same Pod.

In Kubernetes configuration, Pods are not updated but are simply destroyed. New Pods are created when any of the containers are required to be updated.

“Deployments” are something that runs the Kubernetes cluster. They store the configuration, dependencies, resource requirements and access etc. Kubernete also has a “ReplicationController” which allows you to replicate the Pods as per the need.

Kubernetes
Leading cloud providers like AWS, Microsoft and Google provide the managed Kuberenets services viz Amazon Elastic Container Service (Amazon EKS), Azure Container Service (AKS), Google Kubernetes Engine(GKE).

Related Links

Related Keywords

Docker, Container, DevOps, Chef, Puppet

Tutorial