Skip to main content

Two Brains, A Computer, And Lots Of Coffee: How A Pair Of GE Engineers Took NASA By Storm

Bruce Watson
August 02, 2017
Imagine a storm whipping across the Martian desert, blasting everything in its path. When the wind dies down, a robot slowly ventures out of a habitat station and into the dusty aftermath, picking its way around rocks and debris. It assembles a solar panel and adjusts an antenna, providing energy to the station and strengthening its connection to Earth. Then it heads back inside where it detects a leak and patches it, making the room safe for human inhabitants.
The scenario may sound like science fiction, but humans are getting closer to visiting the red planet. When they arrive, they will likely leave behind robots to take care of business while they are away. That's why NASA’s recent Space Robotics Challenge asked programmers to develop the best method for controlling one of NASA’s R5 Valkyrie robots from 140 million miles away. A humanoid robot designed to function in environments that would be deadly to humans, the R5 Valkyrie is sleek, white and armored in plastic.

While the stakes of NASA’s contest weren’t quite life and death, they were still high: The winners would receive $1 million in prizes, the chance to work with actual Valkyrie robots and some fairly stellar bragging rights.

 width= Top: NASA's R5 Valkyrie is a humanoid robot designed to function in environments where Earthlings can't tread. Above: GE Global Research engineers Shiraj Sen (left) and Steven Gray worked nights and weekends to design a method for controlling one of the bots. Their efforts paid off: Their team, called Walk Softly, won second place in NASA’s recent Space Robotics Challenge. Images credit: Walk Softly

The competition attracted 405 teams from 55 countries, including groups from some of the world’s top engineering schools. Some teams had dozens of members and thousands of dollars to spend, but — in the end — the strongest competitors were some of the leanest. The winner, Coordinated Robotics, was a one-man team from Newbury Park, California. Second place went to Walk Softly, a pair of GE engineers from upstate New York.

Walk Softly’s team leader was Steven Gray, a lead robotics engineer at GE Global Research in Niskayuna, New York. The name is a play on Teddy Roosevelt’s famous exhortation to “Speak softly and carry a big stick.” In this case, the “big stick” was the team’s years of experience working with robots. As for speaking softly, the team, which included Gray’s coworker Shiraj Sen, kept a low profile on the competition’s forums, choosing not to boast about their approach.

But Walk Softly’s silence concealed a furious determination. Before moving to GE in May 2016, Gray competed in the high-profile DARPA Robotics Challenge, where competitors tried to create robots that could function in environments that would be deadly to humans. His team didn’t place. “I wasn’t thrilled with that,” he recalls, noting that, when the NASA contest was announced, he felt like it was a chance to try again.

In August 2016, Sen and Gray began working on their submission. They started out with little more than their own skills and knowledge and a lot of coffee, taking over a room in Gray’s house.

The first challenge was to design software that would allow a robot to autonomously push a button to open a door, walk through, and interpret lights on a control panel. In late January 2017, NASA selected Sen and Gray as one of 20 finalists, which meant that they had four more months to come up with a final method for programming and controlling a Valkyrie to complete its tasks in a Martian environment. “We pretty much gave up our weekends,” Sen says.

 width= The challenge's goal was to create an automation or teleoperation system to help the NASA R5 Valkyrie Robot accomplish tasks at a simulated Mars outpost. These tasks included adjusting a satellite dish, setting up a solar array and fixing an air leak in a habitat. Image credit: Walk Softly

Gray and Sen’s day jobs helped them on their journey. At GE, the pair program robots to do maintenance on refineries and other installations where it would be expensive or dangerous to send humans. “Our work experience helped us avoid pitfalls, but we were working full time on multiple projects, so finding the time to work on this was a challenge,” Sen says.

In some ways, that lack of time may have helped them. Rather than trying to completely automate the robot, they only trained it to do some things autonomously — like turning a valve or adjusting its center of gravity. For more complex actions, a human took over operations.

That mix of autonomy and control came out of Gray and Sen’s programming experience. Fully autonomous solutions, in which the robot completely operates on its own, are “brittle,” Gray says, and can quickly deteriorate when something unexpected happens. For example, if a robot is programmed to follow a path that turns to the left, or pick up a briefcase from a table, it can become disoriented if the path turns to the right or the briefcase is missing. This was a crucial part of the NASA challenge, which randomly changed details like topography or the position of key items with every run.

 width= NASA designed its challenge to include variables — an unexpected turn or a missing object, for example — that can confound fully autonomous robot systems. Image credit: Walk Softly

Changing details was only one of many features NASA implemented to ensure that things went wrong often. To begin with, the agency instituted a communications delay of up to 20 seconds, which meant that there was always a distracting lag between the moment the team sent the robot a command and when it was carried out. There was also a wide range in the amount of bandwidth available to the competitors. Sometimes, they were able to send information to their robots at 16 kilobytes per second, but at other times it dropped to an anemic 64 bytes per second. This meant that on occasion, the robot was almost out of their control. However, its ability to conduct semi-autonomous operations meant that Gray and Sen’s robot could continue to function, even when it was out of reach.

Team Walk Softly’s second-place means that they will be able to spend two weeks working with an actual Valkyrie robot. They’re hoping to bring along colleagues from GE Global Research’s robotics and artificial intelligence team to help them test out some of their telerobotics innovations. “We’re hoping that this will lead to more partnerships with NASA,” Sen says.

 width= To make matters even trickier, NASA added a communications delay of up to 20 seconds between the time the team gave a command and the time the robot acted on it. Image credit: Walk Softly

The pair were also awarded $100,000 for coming in second place. “That part’s going to be split evenly between me, Shiraj and the IRS,” Gray laughs. Ultimately, though, a big part of the reward was the chance to connect with other robotics enthusiasts, both fellow competitors and those organizing the competition. “It’s fun to see all the different ways teams approached the same problems," Gray says. "There are so many right answers.”