IBM helps analyse 'space weather' threats in real time

IBM is working with Uppsala University and the Swedish Institute of Space Physics to create a Stream Computing project that can analyse massive amounts of data in real time to better understand potentially damaging “space weather.

By using IBM InfoSphere Streams to capture data from sensors that track high frequency radio waves, endless amounts of data can be captured and analysed on the fly. Over the next year, this project is expected to perform analytics on at least 6 gigabytes per second or 21,600 gigabytes per hour — the equivalent of all the Web pages on the Internet.

Scientists sample high frequency radio emissions from space to study and forecast “space weather,” which is driven by plasma eruptions from the sun. Upon reaching the Earth, such radiation can adversely affect energy transmission over power lines, communications via radio and TV signals, airline and space travel, and satellites.

With the recent advent of new sensor technology and antannae arrays, scientists have begun collecting more information than they’ve been able to intelligently analyse. However, IBM InfoSphere Streams, new software derived from IBM Research project System S, makes an entirely new level of analytics possible by being able to process large volumes of data in real time.

“IBM InfoSphere Streams is opening up a whole new way of doing science, not only in this area, but any area of e-science where you have lots of data coming in from external sources and sensors, streaming at such high data rates you can’t handle it with conventional technology,” said Bo Thidé, head of research at the Swedish Institute of Space Physics and director of LOIS Space Center. “It has helped create a paradigm shift in the area of online observation of the Earth, space, sun and atmosphere.”

Sunspot activity, electromagnetic storms and other types of solar activity can impact communications signals. As critical infrastructure such as power grids and telecommunications networks become more digitally aware, instrumented and interconnected, it is increasingly important to understand how these can be affected by influences such as electromagnetic interference or other changes in the atmosphere.

Researchers at Uppsala University and the Swedish Institute of Space Physics worked with the LOIS Space Center facility in Sweden to develop a new type of tri-axial antenna that streams three-dimensional radio data from space, extracting a magnitude more physical information than any other type of antennae array before. Since researchers need to measure signals from space over large time spans, the raw data generated by even one antenna quickly becomes too large to handle or store.

“We’ve embarked upon an entirely new way of observing radio signals using digital sensors that produce enormous amounts of data,” Thidé said. “With this type of research, you have to be able to analyse as much data as possible on the fly. There is no way to even consider storing it. InfoSphere Streams is playing a pivotal role in this project. Without it, we could not possibly receive this volume of signals and handle them at such a high data rate because until now, there was not a structured, stable way of analysing it.”

IBM’s technology addresses this problem by analysing and filtering the data the moment it streams in, helping researchers identify the critical fraction of a per cent that is meaningful, while the rest is filtered out as noise. Using a visualisation package, scientists can perform queries on the data stream to look closely at interesting events, allowing them not only to forecast, but to “nowcast” events just a few hours away. This will help predict, for example, if a magnetic storm on the sun will reach the earth in 18 to 24 hours.

The ultimate goal of the project at Uppsala University with InfoSphere Streams is to model and predict the behaviour of the uppermost part of our atmosphere and its reaction to events in  surrounding space and on the sun. This work could have lasting impact for future science experiments in space and on earth. With a unique ability to predict how plasma clouds travel in space, new efforts can be made to minimise damage caused by energy bursts or make changes to sensitive satellites, power grids or communications systems.