Big Data Lab

Big Data can be mostly succinctly described as data that is too big, moving too fast, or too unstructured to be managed through traditional computing approaches.  While Big Data challenges are not particularly new, especially to businesses such as GE, highly scalable and distributed, fault-tolerant software systems that enable the cost-effective management and analysis of Big Data have become prominent only within the past decade or so.

The AI team at GE Research leverages Big Data technologies to address digital industrial challenges with respect to scalable data storage, management, and analytic execution.  To that end, the team manages a Big Data Lab of commodity hardware and open source and internally-developed software to enable the storage and processing of a wide variety of data, including time series sensor data, semantically linked data, relational data, images, and more.  The hardware infrastructure includes a 64-node Hadoop cluster with over 24TB of memory and 380TB of direct-attached disk-based storage.  An additional 300TB of network-attached storage is connected to this cluster, resulting in over a half petabyte of data storage capacity for our research applications.  The hardware infrastructure includes two additional small research incubation clusters, including an 11-node and a 6-node cluster, and 2 big memory nodes with 1TB of memory each.
 

A wide variety of open source software is installed on these machines, including Apache Hadoop (HDFS, Hive, YARN), Spark, Cassandra, Kafka, KairosDB, Minio, and more.  We maintain an internally-developed lightweight Big-Data-analytics-as-a-service layer to ease the development and deployment of Big Data applications.  We also maintain semantic triple stores on the clusters and our own internally-developed software to enable the development and execution of knowledge-driven applications, through our open source SADL and SemTK software.