Space weather forecast study turns table of effective predictions on its head
A comparison of solar flare forecasting systems has turned the performance table of apparently effective prediction methods on its head. Researchers at Trinity College Dublin, Ireland, have tested the reliability of seven techniques against their record of predicting flares and non-flare events correctly, as well as their history of missed flares and false alarms. When the predictions were put into context of the Sun’s activity levels over time, some of the most seemingly successful techniques slid down the table. Dr D. Shaun Bloomfield is presenting the findings at the RAS National Astronomy Meeting in St. Andrews, Scotland.
Solar flares are sudden and dramatic releases of energy from the Sun's atmosphere in the form of radiation and electrically charged particles. These eruptions are associated with many aspects of ‘space weather’, which can damage satellites and interfere with communications, navigations and power systems. In our technology-dependent society, accurate advanced warning of solar flare occurrence is an area of increasing concern.
"The most important aspect of any type of forecast is how it performs," said Bloomfield. "If we always say, 'flare expected today', we will have successfully predicted all flares. However, we would be crying wolf and be completely wrong on most days, as flares can occur quite far apart in time. We need be accurate in both our predictions of when flares will occur and when they won’t for this to be of real value to society."
Bloomfield and his colleagues compared seven different systems for predicting solar flares: two of these used images showing the structure of sunspots; the other five used a variety of parameters relating to the magnetic field, including maximum field strength, total flux and strengths of gradients. The source data came from a range of ground based and space based solar telescopes, including SOHO.
In the past, solar flare forecasts have generally been tested over periods of time containing different amounts of flare activity. Bloomfield and his colleagues found that the previous standard system for rating flare forecasts was very sensitive to the relative activity levels between testing periods. The team suggest that the ‘True Skill Statistic (TSS) should be adopted as the new standard rating for solar flare prediction systems. TSS is calculated from the fraction of correct flare forecasts out of all flares observed, minus the fraction of false alarms out of all non-flares observed.
"The benefit of the TSS over other ratings scores is that it is not changed by the number of flares or non-flares observed. We can make a proper comparison of forecast systems, regardless whether they have made 50 or 5000 predictions. Even so, surveys with small data sets are still prone to noise and their results must be considered less reliable,” said Bloomfield.
Bloomfield and his colleagues found that according to the TSS score, adaptive systems including artificial neural networks, did not actually appear to perform much better than simple predictions based on the shape of sunspots. However, the researchers were not able to include several forecasting systems in their study as not all the data needed to calculate the TSS was available.
"If we are to move forward in developing a standard ratings system for flare predictions that produces meaningful results, we need to encourage solar forecasters to be more open about publishing their results. As well as the number of flares correctly predicted, we need to know numbers of correct non-flare predictions, false alarms and missed flares. If these differences in flare statistics are not taken into account properly, some methods can appear to perform better than others when in reality they are the same or worse," said Bloomfield.
IMAGES AND ANIMATIONS
Animation: Zoom-in of the Sun at 08:30 – 10:30 am on 11th January 2013 during a strong flare. The energy to power flares is stored in complex intermingled magnetic fields (top left; black pointing into the Sun and white pointing out) that cause dark sunspot "footprints" (bottom left), as seen on the Sun's visible surface by SDO/HMI. The flare is observed by SDO/AIA as the bright release of light from both the several-thousand degree lower atmosphere (bottom right) and the higher-lying multi-million degree corona (top right). http://www.ras.org.uk/images/stories/NAM2013/5July/bloomfield_flare.mov http://www.ras.org.uk/images/stories/NAM2013/5July/bloomfield_flare_mp4.m4v
Image caption: Zoom-in of the Sun at 9.12am on 11th January 2013 during a strong flare. The energy to power flares is stored in complex intermingled magnetic fields (top left; black pointing into the Sun and white pointing out) that cause dark sunspot "footprints" (bottom left), as seen on the Sun's visible surface by SDO/HMI. The flare is observed by SDO/AIA as the bright release of light from both the several-thousand degree lower atmosphere (bottom right) and the higher-lying multi-million degree corona (top right). Credit: Image produced by D.S. Bloomfield (TCD). Data courtesy of NASA/SDO and the AIA, EVE, and HMI science teams.
Dr D. Shaun Bloomfield
Trinity College Dublin
Tel (office): 353 (0)1-8963281
Tel (mobile): 353 (0)876323789
Dr Robert Massey
Royal Astronomical Society
Tel: +44 (0)20 7734 3307 / 4582 x214
Mob: +44 (0)794 124 8035
Ms Anita Heward
Royal Astronomical Society
Mob: +44 (0)7756 034 243
Ms Emma Shea
Head of Development Communications
University of St Andrews
Tel: +44 (0)1334 462 167
Landline numbers in NAM 2013 press room (available from 9 a.m. to 5 p.m. from 1-4 July, 9 a.m. to 3 p.m. 5 July):
Tel: +44 (0)1334 462231, +44 (0)1334 46 2232
NOTES FOR EDITORS Bringing together more than 600 astronomers and space scientists, the RAS National Astronomy Meeting (NAM 2013) will take place from 1-5 July 2013 at the University of St Andrews, Scotland. The conference is held in conjunction with the UK Solar Physics (UKSP: www.uksolphys.org) and Magnetosphere Ionosphere Solar Terrestrial (MIST: www.mist.ac.uk) meetings. NAM 2013 is principally sponsored by the RAS, STFC and the University of St Andrews and will form part of the ongoing programme to celebrate the University’s 600th anniversary.
Meeting arrangements and a full and up to date schedule of the scientific programme can be found on the official website at http://www.nam2013.co.uk
The Royal Astronomical Society (RAS: www.ras.org.uk, Twitter: @royalastrosoc), founded in 1820, encourages and promotes the study of astronomy, solar-system science, geophysics and closely related branches of science. The RAS organises scientific meetings, publishes international research and review journals, recognizes outstanding achievements by the award of medals and prizes, maintains an extensive library, supports education through grants and outreach activities and represents UK astronomy nationally and internationally. Its more than 3500 members (Fellows), a third based overseas, include scientific researchers in universities, observatories and laboratories as well as historians of astronomy and others.
The Science and Technology Facilities Council (STFC: www.stfc.ac.uk, Twitter: @stfc_matters) is keeping the UK at the forefront of international science and tackling some of the most significant challenges facing society such as meeting our future energy needs, monitoring and understanding climate change, and global security. The Council has a broad science portfolio and works with the academic and industrial communities to share its expertise in materials science, space and ground-based astronomy technologies, laser science, microelectronics, wafer scale manufacturing, particle and nuclear physics, alternative energy production, radio communications and radar. It enables UK researchers to access leading international science facilities for example in the area of astronomy, the European Southern Observatory.
Founded in the 15th century, St Andrews is Scotland’s first university and the third oldest in the English speaking world. Teaching began in the community of St Andrews in 1410 and the University was formally constituted by the issue of Papal Bull in 1413. The University is now one of Europe’s most research intensive seats of learning – over a quarter of its turnover comes from research grants and contracts. It is one of the top rated universities in Europe for research, teaching quality and student satisfaction and is consistently ranked among the UK’s top five in leading independent league tables produced by The Times, The Guardian and the Sunday Times. The University is currently celebrating its 600th anniversary and pursuing a £100 million fundraising campaign, launched by Patron and alumnus HRH Prince William Duke of Cambridge, including £4 million to fund the creation of an ‘Other Worlds’ Think Tank and Observatory. The new think tank and Observatory project will extend the University of St Andrews’ flagship work on extra-solar planets, and provide a creative environment for problem-focused research, education and continuing public engagement. For further information go to: www.st-andrews.ac.uk/600/