|
Post by curiousgeorge on Apr 3, 2010 13:19:12 GMT
A question: Since the term "Reliability" often comes up in discussions of climate modeling I'm wondering if anyone has subjected this type of software, and therefore it's output, to a formal reliability study? I would like to see any metrics associated with this, in the hope that it would resolve at least some of the ambiguity associated with the question: "How reliable are climate models (quantitatively)"? In my view it is insufficient, and perhaps even decieving, to merely claim that a climate model is (or is not) "Reliable" without such metrics. The following describes a general approach and some of the issues that should be addressed: www.ece.cmu.edu/~koopman/des_s99/sw_reliability/
|
|
|
Post by hunter on Apr 3, 2010 15:14:53 GMT
Great question. The reviews done of the code released by the climategate leaker indicate that the code is crap. I will dig around for some links. Since the code is written in many cases by the same people who collect, massage and destroy the original data, the really good bits will be difficult to dig up. Since we know that the CRU data is crap, and most temperature products correlate closely to it, I think it is not unreasonable to assume that the rest of the temperature products are crap as well.
|
|
|
Post by msphar on Apr 3, 2010 16:19:46 GMT
It is my understanding that several of the GCMs are used every year to project short term events in the tropical Atlantic to track and predict tropical cyclones. I watch this area carefully as I have a boat down there that I care deeply about. Typically the data gets reported and analyzed by lay people who are focused on cyclone genesis. As I have watched this activity for more than half a decade, I have detected no improvement in the GCM skill level. They are good for about 2 to 3 or 4 day forecast but quickly their predictiveness falls apart beyond those horizons. Not one GCM has stood out for longer than a season. So consistency is not their strong suit either. Every year I hope for better results. So I am cheering for them but every year I fall back to seat of the pants viewing of satallite data for self interest. If you can't trust the GCM for more than a half dozen day forecast, thinking in terms of multidecadal forecast seems ludicrous in the extreme.
|
|
|
Post by curiousgeorge on Apr 3, 2010 16:31:13 GMT
Great question. The reviews done of the code released by the climategate leaker indicate that the code is crap. I will dig around for some links. Since the code is written in many cases by the same people who collect, massage and destroy the original data, the really good bits will be difficult to dig up. Since we know that the CRU data is crap, and most temperature products correlate closely to it, I think it is not unreasonable to assume that the rest of the temperature products are crap as well. I do appreciate your interest and whatever help you can offer, since I'm retired and not active in technical fields any longer. Just as a point of interest the following quote from the above url is particularly important ( I feel ) given the potentially massive social and financial impact that political, regulatory, and economic policy decisions will have as a direct result of the output of such models. We do not tolerate software problems in our vehicles, or even in simple video games, so I see no reason why formal reliability metrics should be ignored in something as important as this. Thanks again.
|
|
|
Post by socold on Apr 3, 2010 21:46:55 GMT
A question: Since the term "Reliability" often comes up in discussions of climate modeling I'm wondering if anyone has subjected this type of software, and therefore it's output, to a formal reliability study? I would like to see any metrics associated with this, in the hope that it would resolve at least some of the ambiguity associated with the question: "How reliable are climate models (quantitatively)"? In my view it is insufficient, and perhaps even decieving, to merely claim that a climate model is (or is not) "Reliable" without such metrics. The following describes a general approach and some of the issues that should be addressed: www.ece.cmu.edu/~koopman/des_s99/sw_reliability/ Interesting question, but what does climate model reliability mean in a practical sense?
|
|
|
Post by sigurdur on Apr 4, 2010 3:31:31 GMT
If the reliability of the models is poor, yet governments kill massive amounts of people through their response to the climate models....I would say in a practical sense it becomes extremely important.
|
|
|
Post by icefisher on Apr 4, 2010 5:49:34 GMT
Interesting question, but what does climate model reliability mean in a practical sense? As General Dwight D Eisenhower said on June 5, 1944 after reviewing the weather forecast: “I hope to God I know what I’m doing.”
|
|
|
Post by socold on Apr 4, 2010 12:39:21 GMT
But that sounds like you are talking about model accuracy, not software reliability. What does reliability in context of a climate model? This is all beginning to sound a lot like the empty rhetoric of management-speak to me.
Next we might discuss the synergy of climate models. Has it been measured? If not how can management trust them?
|
|
|
Post by curiousgeorge on Apr 4, 2010 13:50:39 GMT
But that sounds like you are talking about model accuracy, not software reliability. What does reliability in context of a climate model? This is all beginning to sound a lot like the empty rhetoric of management-speak to me. Next we might discuss the synergy of climate models. Has it been measured? If not how can management trust them? I suggest you do a little reading about software reliability/quality control. It is a well recognized field, complete with International and National Standards that specify configuration management, testing, etc. protocols. It is most assuredly not a "empty rhetoric of management-speak". It is a requirement in Government contracts and in regulated industries such as aerospace, as well as common practice for the overall software industry - including Microsoft, believe it or not. An example: Excel, which is used by many people for data analysis and manipulation, has been known to contain calculation errors of various severity. Those errors may, or may not, be obvious to the user, will often not cause a "crash", but can lead the user to erroneous conclusions or cause "downstream" errors in subsequent calculations. The same can be said of many other commercial software/modules which may be used in conjunction with customized, task specific software written in-house. This is not a trivial issue.
|
|
|
Post by socold on Apr 4, 2010 14:56:13 GMT
I am sure it is a well recognized field, but so far noone has suggested any metrics that could be applied to climate model software to quantify "reliability". An example suggestion might be "measure the number of bugs discovered versus lines of code modified" or something. What I am asking for is what people are suggesting measuring "reliability" of climate models practically means. Surely without that this discussion won't go anywhere?
|
|
|
Post by curiousgeorge on Apr 4, 2010 15:12:30 GMT
I am sure it is a well recognized field, but so far noone has suggested any metrics that could be applied to climate model software to quantify "reliability". An example suggestion might be "measure the number of bugs discovered versus lines of code modified" or something. What I am asking for is what people are suggesting measuring "reliability" of climate models practically means. Surely without that this discussion won't go anywhere? Perhaps this will help you: www.easterbrook.ca/steve/?p=1558Be sure to read Steve Easterbrook's bio in case you have questions about his qualifications. There is a great deal of information on his site that pertains to this specific issue.
|
|
|
Post by magellan on Apr 5, 2010 2:45:11 GMT
I am sure it is a well recognized field, but so far noone has suggested any metrics that could be applied to climate model software to quantify "reliability". An example suggestion might be "measure the number of bugs discovered versus lines of code modified" or something. What I am asking for is what people are suggesting measuring "reliability" of climate models practically means. Surely without that this discussion won't go anywhere? Perhaps this will help you: www.easterbrook.ca/steve/?p=1558Be sure to read Steve Easterbrook's bio in case you have questions about his qualifications. There is a great deal of information on his site that pertains to this specific issue. NASA has had a software V&V requirement for many years. Why hasn't it been applied to climate models? Hmmm? www.goes-r.gov/procurement/flight_documents/NASA-STD-8739-8-1.pdf
|
|
|
Post by socold on Apr 5, 2010 2:53:20 GMT
How do you know it hasn't?
|
|
|
Post by slh1234 on Apr 5, 2010 3:50:31 GMT
I am sure it is a well recognized field, but so far noone has suggested any metrics that could be applied to climate model software to quantify "reliability". An example suggestion might be "measure the number of bugs discovered versus lines of code modified" or something. What I am asking for is what people are suggesting measuring "reliability" of climate models practically means. Surely without that this discussion won't go anywhere? In the early parts of design, discussions of "what does this mean" are hammered out. If you're a participant in such discussions, you are expected to help define that instead of just saying "we don't know yet what we're trying to define." So do you have a suggestion on how to define it? What would you suggest? Surely you think some method of measuring reliability should be established? So help us understand what you would think would be helpful.
|
|
|
Post by icefisher on Apr 5, 2010 6:15:11 GMT
I am sure it is a well recognized field, but so far noone has suggested any metrics that could be applied to climate model software to quantify "reliability". An example suggestion might be "measure the number of bugs discovered versus lines of code modified" or something. What I am asking for is what people are suggesting measuring "reliability" of climate models practically means. Surely without that this discussion won't go anywhere? In the early parts of design, discussions of "what does this mean" are hammered out. If you're a participant in such discussions, you are expected to help define that instead of just saying "we don't know yet what we're trying to define." So do you have a suggestion on how to define it? What would you suggest? Surely you think some method of measuring reliability should be established? So help us understand what you would think would be helpful. Exactly! And that spells the difference between the space program that sought out people that could get jobs done and the casualness of using academia to address global warming. What these people don't realize is if you miss the moon by a 100,000 miles you get your budget pulled and nobody gets to the moon for another generation or two. If AGW is real and you miss the mark embarrassingly. . . .such incompetence sets back science for a couple of generations. As they say haste makes waste. What we have been seeing is a lot of people getting paid a lot of money with no expectation of the work being done in a workmanship manner. Briffa's and Mann's work is just plain shoddy.
|
|