Student Projects

OpenStack Projects

“Thank you to the IAP, CMU and Cornell teams for their hard work and attention to detail.
Outstanding collaborations with fantastic results!”
– Dr. Brad Topol, IBM Distinguished Engineer, OpenStack

Openstack Carnegie Mellon University IBM

2016 Cornell OpenStack Team (left to right): Qin Jia, Dr. Weijia Song, Vishnu Govindaraj,
Mengcheng Qi, Xiao (Alex) Tan, Samir Borle, Richard Quan, and Prof. Robbert van Renesse.

“This was a great opportunity for us to learn how to work in the open source community. Very complementary to our coursework, teaching us industry best practices for test-driven software development, and being held accountable for high quality contributions to OpenStack.”

                                      – Alex Tan, Cornell Computer Science, Master of Engineering

The Cornell OpenStack team was formed after the IAP Cornell Cloud Workshop in October 2015, where Kiloeyes was described in a presentation to the attendees by IBM Senior Engineer and OpenStack Contributor Tong Li. A special class was created for the students to earn credit in the Spring 2016 semester, led by Professor Robbert van Renesse.  Prof. van Renesse is an expert in distributed systems, particularly in their fault tolerance and scalability aspects, chair of ACM SIGOPS, editor for ACM Computing Surveys, and an ACM Fellow.  The team contributed to the Kiloeyes service, an open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution. It uses a REST API for high-speed metrics processing and querying, and has streaming alarm and notification engines.

Tong Li (left), IBM, Senior Software Engineer and OpenStack Contributor

describes Kiloeyes at the IAP Cornell Cloud Workshop in October 2015. Tong pioneered and leads the development of Kiloeyes, the OpenStack cloud monitoring solution, and is currently building out new micro services for Kiloeyes to reliably analyze real time message streams of several hundred thousand per second.  Tong met informally with several students at the Workshop during the breaks and networking session to share his perspectives and experiences as a software engineer at IBM, focused on open source solutions.  (See the summary of the 2015 Cornell Workshop).

Kiloeyes uses Kafka as its message queue, to reliably consume over >>100,000 messages per second, that are distributed to data processing engines for alarms, anomaly prediction and notifications. Elasticsearch is the back end data store, providing fast and durable storage of all incoming messages at the pace delivered by Kafka. The basic architecture is shown in the figure below.

Kiloeyes Architecture: The Analyzer/Transformer/Aggregator represents multiple mico-service engines that operate independently, such that new engines for specific data processing and analytics tasks can be easily added.

The Cornell team collaborated together on three important tasks specified and mentored by IBM Senior Engineer and OpenStack Contributor Tong Li: (1) Security, (2) Metrics visualization, and (3) Kiloeyes integration with the other OpenStack services via support of the OpenStack Ceilometer API.

Security: The integration of authentication and security features into Kiloeyes required changes to the Kiloeyes server pipeline. The approach was to authenticate incoming requests against OpenStack's Keystone service, by integrating the Keystone Middleware project into the server pipeline. Then, user authentication information was appended to the Kiloeyes metrics, so every metric contains data about which tenant, service, and user posted it. Finally, the added authentication data was used to filter queries to Kiloeyes such that only metrics that a user is authorized to view would be presented.

Ceilometer API: Ceilometer is a legacy Openstack monitoring service that was recognized not to scale well for large clusters; indeed, this limitation was the motivation for the development of Kiloeyes. However, the Ceilometer API is a natural and convenient means to interface Kiloeyes with the other Openstack services, hence the goal was to the implement the Ceilometer API on Kiloeyes based on Ceilometer API's documentation. The team successfully implemented Meters API including Get Meters, Get Meters by Name, Get Meter Statistics and Post. Additionally, they built a parameter search utility in Meters that supports all Ceilometer query and aggregation to enable users to query the metrics they desire.

Visualization: Kiloeyes lacked any Visual Representation of the aggregated metrics. This limitation was overcome with the help of the open source tool Kibana. The team successfully integrated Kibana with the elastic search host and created Graphs and Charts representing the Kiloeyes metrics. A dashboard plugin was created for Horizon, the Openstack dashboard application and the visualizations were added to the dashboard. The team also made sure that the details represented in visualization can change according to the project selected by the end-user. Below is a screen shot of Network packet activity collected by Kiloeyes and displayed on the Horizon Dashboard.

Kiloeyes Visualization: Network packet traffic vs. time is aggregated by Kiloeyes and plotted in the Horizon dashboard shown above. The tabs in the panel on the left of the screen shot (“Load Avg”, “Process Thread Count”, etc.) have several standard line and pie charts.  New metrics and charts
can be easily customized by an OpenStack user, and added as a tab.

The Kiloeyes code is available on github (

The Cornell project built on top of the outstanding work performed at Carnegie Mellon in 2015. The CMU project culminated in the first availability of Kiloeyes in the OpenStack Liberty release in October 2015.

Graduate students Han Chen, Shaunak Shatmanyu, and Yihan Wang and CMU research scientist Dr. Ipek Ozkaya at Carnegie Mellon University in Pittsburgh collaborated with IBM Senior Engineer and OpenStack Contributor Tong Li on a two-semester project to build out the python-monasca project, including new data processing engines for alarms, anomaly prediction and notifications. The students merged their first production code to Stackforge at the mid-point of the project in May 2015, and final code in August 2015. Python-monasca was migrated to become OpenStack Kiloeyes in August 2015, and released with Liberty in October 2015. Hence, this project is a significant achievement for the students and IBM, and a noteworthy milestone for OpenStack.

OpenDaylight Projects

OpenDaylight (ODL) is an open platform for network programmability to enable software defined networking (SDN) for networks at any size and scale. The “Internet of Things” IoT Data Management (IoTDM) project uses the ODL infrastructure to provide a data-centric middleware that acts as an IoT data broker, and enables authorized applications to post and retrieve IoT data uploaded by any device -

The IoT data is organized in a massive resource tree (potentially millions of nodes). This tree contains measurements from devices (the things) and its associated attributes.  The attributes represent metadata about the resource, for example access rights, creation time, children list, owner, size, quota, etc.  IoTDM complies with the oneM2M standards for machine-to-machine and IoT communications -

Boston University (BU) students built apps for Cisco’s Smart Cities projects to provide visualizations of mobile device positions on interactive maps of both indoor facilities and aerial/satellite imagery by combining signals acquired from disparate “things” including Bluetooth low energy beacons, internet routers, and GPS.

Using the BU apps, mobile users can (a) search for other registered users on their mobile devices, (b) view a “heat map” that displays real­time “people traffic” and historical data, and (c) manage the Tree data using a unique data visualization tool to search, add, edit or delete data (“leaf”) nodes for users, devices, locations and maps visually displayed on a browser.

Students who contributed to the apps include Mark Barrasso, Jose Bautista, Justina Choi, Sean Liu, Nehal Odedra, Niklas Kunkel, Qingqing Li, and Yingchao Zhu.  The students were supervised by Prof. Orran Krieger and Dr. Ata Turk of the BU Department of Electrical and Computer Engineering, and Cisco Principal Engineer Lionel Florit, who scoped the projects and guided the students in weekly meetings.

Mark Barrasso won the Top Hacker Award for the Smart Cities projects, for leadership in application design and coding. Mark also acted as scrum leader for one of the BU student teams, and represented BU and his colleagues at Cisco Live in San Diego in June 2015.

Project demos presented at Cisco Live are here.

Mark Barrasso (center) at Cisco Live in San Diego on June 10, 2015, is presented the Top Hacker
Award by Lionel Florit, Cisco Principal Engineer and Kristen Wright, Director, Cisco Research Center.

ARM-based Cloud Servers

Cavium has entered the data center server market with the 48-core ARM-based ThunderX processor -

Cornell graduate student Xiaodong Wang and Professor José F. Martínez are working with Cavium Distinguished Engineer Bryan Chin and Bobbie Manne on the ThunderX platform. They are studying performance scalability within power bounds, and limitations to scalability both in hardware and software, and mechanisms to mitigate identified limitations.

Cornell PhD Student Xiaodong Wang in his lab at Cornell with Cavium board.