Excel 2010 Powerpivots are great for exploring large datasets like the HHP. Because the have an OLAP engine & Data compression (Vertipaq) under the hood, they gobble up these big datasets. The slicer functionality means it quick to slice & dice the dataset in multiple.

I have managed to get php and mysql working well now. I want to use this competition to improve my programming skills on a difficult task.

I have been learning programming for 6 months now and I am improving.This is definitely a difficult task or else the problem would have been solved years ago-think of all the money governments put into health research.

I am a novice at programming but not at stats or biology so that is why  I think it is worth having a go.

And no one is using Excel??? Is there free Minitab?

I'm using Mathematica V8 ... anyone know if this is legal?

I'm using Haskell. Yes, I'm brave.

1) SQL ... 2) Excel 3) Rapid Miner 4) R

Rapid Insight Analytics for modeling. Rapid Insight Veera for preparing the analytic file and scoring.

You can always try out Oracle Express, should work on your laptop and will have no issues digesting the large amount of data being used in the competition....free download from Oracle.com

I tried oracle express and it was too slow at executing queries!!

Will continue with mysql and php together.

Thanks for the suggestion though.

Seriously though - is Mathematica an acceptable product to use in this competition? You have to get a license to use it, but there's no restrictions as to who can get a license and it isn't prohibitively expensive. Thanks ahead of time for the clarification.

Is anyone having memory problems getting anything done with the claims data in R, particularly in Revolution?

yes, on my laptop (it has only 2gb ram). but I'm going to switch to my pc as working platform for this competition as soon as I get to install win7 64bit on it.

You need to use vector operators instead of loop constructs. If memory is still a problem you should move to linux and 64 bit versions of R or try using biglm. I have tried 32 and 64 bit XP with R, but available memory is still strongly limited.

 

What opperations are you trying to do?

 

I'm been testing out Revolution for the past 2 weeks.. it's not bad but still needs lot of work to be a good IDE. Some things drive me nuts in it.. I would of prefered if they built it on top of Eclipse rather than V.Studio..

I have Win 7 64-bit with 4Gb of RAM and everything I do on the claims data seems to be fine. Doing a random forest, I can only use about 200 trees.. if I try 300 or more I run out of memory.

Do the rules prohibit using a tool such as Oracle Data Miner?