My colleagues Mark Grabb, who leads our analytics team, and lead scientist Ben Beckmann spoke recently at a meeting of the New York City Chapter of the Data Warehouse Institute (TDWI NYC). Their wide-ranging presentations offered a fascinating perspective on the progress we’ve made in areas such as advanced analytics, machine learning, software-defined networks, digital manufacturing, predictive maintenance, and the Industrial Internet.

The pace of progress in those areas is simply astonishing, and we’re only in the early innings of the game. Mark and Ben had been asked to talk about Hadoop, NoSQL, and other kinds of non-traditional data management frameworks, but the conversation quickly moved beyond the topic of database architecture and expanded rapidly to encompass many of the other cool areas we’re pioneering at GE Software. 

For example, as we push outward on the boundaries of the Industrial Internet, we’re beginning to see the limitations of existing information networks. More connected machines and devices results in more data flowing over more channels. In his presentation, Ben spoke about Claude Shannon, the “father of information theory” and the man who ushered in the modern digital age. According to Shannon, there’s only so much data you can push across a channel before you start losing information.

Should we care about Dr. Shannon’s theories today? I think we should, because we’re coming perilously close to hitting theShannon Limit – a barrier that would prevent us from adding more information to existing systems.

There are techniques that enable us to maximize the flow of information across fiber optical networks, but even our mastery of those techniques won’t allow us to simply jump across the barrier when we hit it. When will we reach the Shannon Limit? A lot depends on how fast commercial information traffic grows. If it grows at 20 percent annually, we’ll hit the wall in 2021. If it grows at 60 percent annually, we’ll hit the wall next year. Building more infrastructure will push the wall further out. If we’re lucky, the Shannon Limit will always remain a few years ahead of us. If not, the world will have to cope with slower and less reliable flows of information. Hitting the wall would mean fewer opportunities for all of us, since so much of our economy depends on the rapid and reliable flow of digital information.

In his presentation, Mark talked about some of the newer trends in data science and advanced analytics. One of the trends, he says, is “humans moving higher up in the food chain.” Automation isn’t pushing us out of the picture; it’s enabling us to focus on higher-level tasks. Humans will always be in the loop, says Mark, and we’ll be using analytics to help us make better and wiser decisions about managing our resources.

For me, that’s the real promise of the exciting new technologies we’re pursuing:  making smarter choices and preserving our world for future generations.