AI weather

are we having another "alpha go moment"?

forecasting weather is notoriously difficult. it’s the realm of hours and days long calculations on powerful supercomputers. and it’s still not really that great. runs of climate models take many thousands of processor hours. some of these projects get allocated in millions of node hours.

but might that be about to change?

because this could be a VERY big deal and really underlines just what a game changer AI/machine learning can be.

LINK

Artificial-intelligence (AI) firm Google DeepMind has turned its hand to the intensive science of weather forecasting — and developed a machine-learning model that outperforms the best conventional tools as well as other AI approaches at the task.

The model, called GraphCast, can run from a desktop computer and makes more accurate predictions than conventional models in minutes rather than hours.

this has a level of outlandish potential that’s hard to even wrap your head around at first. and the speed is just a part of it.

The trained model uses the ‘current’ state of global weather and weather estimates from 6 hours earlier to predict the weather 6 hours ahead. Earlier predictions are fed back into the model, enabling it to make estimates further into the future. DeepMind researchers found that GraphCast could use global weather estimates from 2018 to make forecasts up to 10 days ahead in less than a minute, and the predictions were more accurate than the ECMWF’s High RESolution forecasting system (HRES) — one version of its NWP — which takes hours to forecast.

“In the troposphere, which is the part of the atmosphere closest to the surface that affects us all the most, GraphCast outperforms HRES on more than 99% of the 12,00 measurements that we’ve done,” says computer scientist Remi Lam at DeepMind in London.

this has some surprisingly earth shaking potentialities, especially as this is still very new, gen 1 stuff. it will rapidly improve. this is exactly the sort of task that self learning AI can excel at. there is a staggering amount of data, it’s linked and cross linked and back chained in horrifically complex ways, and it’s frankly a complexity level that is just plain beyond humans. our existing systems are brute force attacks trying to model cubes of air that accumulate error rates at great speed while eating ever more computational power.

but suddenly minutes of desktop time can outperform hours of a supercomputer.

really stop to think about what that means.

stop to realize that this is an “alpha go moment” where once more, machine learning is showing us that we never even understood the rudiments of a game we’re been playing for ages. we thought we were masters. we did not even know the basics. we did not even understand the moves the machine made until it mopped the floor with what we could conceive. top players laughed at its openings as senseless, then lost. bigly. this problem space is too big for humans to model or even comprehend in any meaningful way. we just take stabs in the dark and hope to get lucky.

i think people get confused in these ideas of “fuzzy logic” and “creativity” and “human equivalent intelligence” and lose sight (perhaps because it’s such a narcissistic blow to us) of the fact that for a great many tasks, human intelligence is insufficient or even inapplicable to the point of trying to drive nails with a wrench or remove screws with a hammer.

we seem to be in the reflexive “AI is dumb, look, it misreported these research studies or it cannot count these dogs in a group” but this loses sight of the things at which it’s astonishing. we can pick fields where humans are still better, but the number where AI excels will keep rising. this genie is not going back in the bottle, especially in complex systems.

gatopal™ doomberg was talking the other day about how rainfall in sichuan drives coal prices in appalacia. this is one of those “no way” associations that seems like it must be nonsense but that is, in fact, both true and clearly causal. sichuan has huge hydro electric plants. that water becomes electricity when it runs through dams. that reduces the demand for coal for coal fired power plants that likely would have been imported. it’s obvious once you see it. would you have ever thought of it?

how many millions of these are lying around having never been noticed?

this is the sort of stuff that’s trivial for AI and it can apply, with outlandish scope and without bias, self-designed learning models to massive datasets. the self design aspect is a very big deal. many people seem mired in the past paradigm of “you have to tell the computer what to do” as though humans are somehow the ones spotting the relationships and setting up the logic structures. they aren’t. this is something new. this is something alien.

it’s teaching itself how to learn, then what to learn.

it’s going to crack open all sorts of previously unapproachable problems.

i have friends working on this for the endocrine system (one of the great messes and mysteries of the human body that we are currently essentially attacking with flint axes). they do not propose to tell AI how it works, they going to ask AI to tell them.

design of transport, logistics, and supply chain will all change.

AI might well solve the traveling salesman problem.

and this is just the beginning with tiny amounts of rude data.

imagine what this weather system will be able to do when it becomes massively distributed and can take temperature and pressure and humidity data from a billion cell phones and computers all linked together and talking to each other to solve problems.

imagine how it could reshape the presumptive fantasy-land of current climate models which evidence so little predictive power despite being run on some of the most powerful computers in academia (and government).

we likely have no idea how many of these pieces even move on the board. but perhaps AI will. (this likely has many current climate grant recipients quite worried as their shiny toys and unearned authority may be about to be supplanted)

i have lots of doctor friends who are sure that AI will never replace them, but i fear they are in for a real shock and i suspect that even with all the guild backlash, 10 years from now the idea of allowing a human to diagnose you will be hilarious.

the pace is going to astonish.

less than a year ago, this was an AI cat.

today you can generate the one below in 30 seconds from a 2 line prompt.

make no mistake: