The Tengu Platform allows Fed4FIRE users to easily set up an experiment for big data analysis and cloud applications.
The platform currently is built around a RESTful API that is interpreted in order to define which type and number of resources are needed in the experiment. Tengu uses an incredible amount of different technologies (the supported set of technologies is still growing). Trying to deploy and configure these big data and cloud experiments is very cumbersome – especially the first time. It is exactly this problem Tengu solves for the experimenter.
Tengu comes in many shapes and flavors. To understand this, it is best to have a glance at Tengu’s core architecture.
The overall Tengu architecture has two distinct stages. The first stage consists of the front end, which translates RESTful method calls from the experimenters into an RSpec of a new Tengu core instantiation. This created RSpec is deployed by jFed on the selected Fed4FIRE testbed. Once the deployment is finished, the second stage starts. The configuration management software makes sure all different components are correctly configured and interconnected, resulting in a new full-featured Tengu core setup. We use Chef to manage the configuration of the different components.
The experiment that is deployed by default, is the Tengu core setup. This deployment, an adaptation of the Nathan Marz’s Lambda Architecture, includes all currently available Tengu features. The main features are:
- support for lambda architecture analysis (in short lambda jobs);
- batch analysis via Apache Hadoop;
- stream processing via Apache Storm;
- application virtualization via a private OpenStack cloud;
- support for the following Tengu managed data stores: MySQL, ElasticSearch and Cassandra (more coming soon);
- live data store transformation.
We understand that not every experiment will need this many features and is, for example, only interested in setting up an Apache Hadoop cluster. Currently, it is also possible to set up the individual components by using the Tengu RESTful API. In this way it is possible to setup:
|an Apache Hadoop cluster||
an Apache Storm cluster
|and a private OpenStack cloud|