« Looming Decisions | Main | 1 Nation - 11 Regions »
December 10, 2006
Terabyteosaurus
I know I haven't blogged in awhile, but I'm still alive. I was ambushed by the terabyteosaurus (hat tip to my brother Nathan for coining the term, I like it), but I've managed to escape for the time being. Joel's/my codes have been happily chugging away (finally) both on the CAPTEX (Cross-Appalachian Tracer Experiment) and IHOP (International H2O Project) data I'll be using for my thesis. That's a very good thing since I'd really love to have some mid-stage CAPTEX results to show in my Meteo 580 paper/presentation, and I really need to get the IHOP stuff done for Joel & Sue before DTRA (Defense Threat Reduction Agency) comes up for the IPR (interim project review) on 19-20 December. (I also will probably end up having to give a modified, more technical version of my 580 presentation to DTRA.)
So anyway, back to the terabyteosaurus. Running all these codes has been generating a tremendous amount of data, so when I tried to set up another code run on Friday night after getting back from trivia at the G-Man, it failed right away, with a message telling me the disk was full. Gulp. So yeah, I'd generated nearly a terabyte of data, which had sucked up 100% of the available space on that partition of the zeus raid (literally, not a single byte of free space was left). And that's really not good, since the operational version of MM5 runs on zeus... So I sent a somewhat-panicked email after midnight on Friday night to Joel, Sue & Chuck to inform them of the situation and that I was dead in the water until and unless they could find me more disk space somewhere. Fortunately Chuck was able to give me access to jupiter (another raid in the meteo dept) in short order, complete with nearly 3 terabytes to play with, so yesterday I spent awhile getting all my stuff transferred to there. Now I just need to try to figure out if 3 TB is gonna be enough space for everything I'll eventually need. For my immediate needs though, I should be all set. Hopefully I can get everything finished so that I can have some pretty graphics and results to show in a week.
I went to see "Borat" with a few of the other meteo grads this past Thursday. I was really looking forward to it because I'd heard some good reviews, and I was also curious just because of all the cultural hype and whatnot. I must say I came away from it fairly lukewarm though. Parts of it were extremely funny (even if they were racist/offensive/explicit), and I really liked the first half of the film. But the second half really started to drag and get old. That and the whole hotel wrestling scene lasted way too long, and really wasn't necessary at all... All in all I'm glad I saw the movie, but it's one of those movies where seeing it once is definitely enough, I think.
Senator Inhofe from Oklahoma and the U.S. Senate Committee on Environment and Public Works have come out with a document, "A Skeptic’s Guide to Debunking Global Warming Alarmism. Hot & Cold Media Spin Cycle: A Challenge To Journalists who Cover Global Warming." (Click here to download the pdf.) I'd encourage y'all to check it out.
It's hard to believe that Christmas is only two weeks away, particularly when we're having temps in the 40s and low 50s. I'd better get a move-on in the Christmas shopping department, if I wanna make sure stuff all arrives on time.
Back to doing something productive. Like going to sleep so I can wake up relatively early and get work done. I'm also doing the forecast on air on CNET tomorrow afternoon (for the last time?), so I'll be on Channel 7 here in State College at 5:55pm, in case you feel like taking a look.
Posted by Jared at December 10, 2006 11:42 PM
Comments
Whoa! You have a program that chugged out a whole terabyte of info? Is it video? Did you program it yourself? If so, in what? Do you know how much info that is? If that's binary info I'm going to be super impressed Jared. I'm going to be even more impressed if you write an algorithm that will actually analyze, process, or even visually display that amount of info. Excellent work, I will see you soon.
Posted by: dave at December 12, 2006 01:19 AM
It isn't a single program really, although all these codes could certainly be made into modules in one giant, scary Fortran90 code if I wanted to. Also, as a disclaimer, my advisor had already written most of the code, I only modified and tweaked some of it and then figured out how to use it.
The data we're generating isn't video, but rather wind and contaminant concentration data written out at 25 vertical levels above every grid point on a very large domain (Great Lakes & Northeast) that is at a 4-km resolution. It's written out every 15 minutes for a 24-hr span. And then multiply it by 19 members in the forecast ensemble of the wind field that we're using currently. Then we're taking variances and covariances of a whole host of variables across the entire domain and across the entire ensemble. And since this entire mess of data is in ASCII and not binary, it can be really easy to see how that all would total about a terabyte of data. :-)
For visualizing all this stuff, I'm using TecPlot 10, a cumbersome but powerful plotting package that makes really nice-looking plots.
I hope that helps shed some light on what I'm doing. See ya at the end of the month!
Posted by: Jared at December 12, 2006 11:21 PM