The papers written by the milestone winners are now available here. As described in section 13 of the rules, if you have any concerns about these papers, you have 30 days from their posting to provide your feedback.

Thank you very much. My concern is that there is no way for us to test the Prediction Algorithm. According to Rule 13: Sponsor will deliver the Prediction Algorithm and documentation to the judges and also post the information on the Website for review and testing by other Entrants.

This is not a complaint relating to conditional winners' methodologies. I'm just confused that the information provided is very vague and there is no Prediction Algorithm posted for us to review or test. Thank you.

I don't have a concern, but I do have a question, as I find Rule 12 to be a little ambiguous. The way I read it ... disclaimer, not a lawyer! ... it requires the candidate winners to submit algorithm code that is CAPABLE of being verified as having produced the entrant's best submission file. I'm not sure the Rule equally binds Kaggle to then actually DO the verification process to replicate the results, if it appears to Kaggle's experts that everything is on the up&up.

I appreciated reading both papers. Both met my expectations for good faith sharing.

Market-Makers's paper includes, as Appendix A, a script "to produce a model capable of a leaderboard position in the top 10% of teams." In other words, that's the outline, but not ALL the detail, of what they've done so far. There could even be an undisclosed breakthrough that accounts for the distance between a 90th-percentile score and the(ir) best score to date.

Willem Mestrom used a stochastic gradient descent technique, which I know absolutely nothing about, so I didn't have much in the way of immediate take-aways from that one at first reading. This paper stuck more to a conceptual description, and didn't offer a run-able script, or as much clarity in the data preparation.(My ignorance of the technique used here could be handicapping my understanding of the data set-up.)

Neither descriptive paper presented enough information to other other competitors that would allow us to exactly replicate the results of the milestone winners ... which is perfectly fine, because the Rules didn't say that was a requirement.

But did the Kaggle Judges get more detailed scripts that they ran, and in each case, successfully replicated the cited results ?

Again, I don't think it's absolutely mandatory under the rules that a re-run based verification happened for the MileStone Prizes. I'm just curious as to whether it did.

(For the Final Prize, I certainly would expect complete verification, even if running all the models makes somebody's biggest CPU scream continuously for a month and glow red in the dark. Be a good place for a web-cam: Here's the Kaggle Verification Machine working hard, glowing in the dark and smokin'. Something to look forward to.)

Here's to you, MileStone One Champions! [raises mug in air] ... you do know you're going to have to run hard to stay ahead, right ?

 

 

 

 

I have to say I am appalled that people are getting high scores using trial and error with standard algorithms without really understanding why they work! Where is the science in all this data mining/analysis?

This is fine with me, whatever work works. What concerns me is that they could have as well written that they tried random numbers and picked random numbers that work best. Or that judges picked us because they like us. Either way we can't verify any of that.

How do we provide feedback? Through the forums?

I'm not sure I understand .... are you suggesting our leaders (or others?) are merely throwing monkey-see-monkey-do things at the data, with no attempt to understand the data or the business problem, or are you bemoaning the fact that machine learning algorithms allow people (data scientists, loosely) to approach problems without necessarily making (as many) a priori assumptions about what are the influential drivers of complex and poorly understood relationships in that noisy data ?

If it was really easy, there'd be a 10,000-way tie for 1st place.

I've got the following email ealier, but I suppose providing feedback through the forums would work as well.

Heritage Health Prize Administrators support@kaggle.com via messagingengine.com to me

show details 1:31 PM (7 hours ago)

Dear Sarkis,

The preliminary winners of the Heritage Health Prize Round 1 Milestone prize were the "Market Makers" team in first place and Willem Mestrom in second place.

Per the contest rules, progress prize winners were required to submit a paper describing their technique. We have posted these papers on the Milestone 1 leaderboard at

http://www.heritagehealthprize.com/c/hhp/Leaderboard/milestone1

As described in section 13 of the rules, if you have any concerns about these papers, you have 30 days from their posting to provide your feedback.

Please provide any feedback by replying to this email.

Thanks,

Heritage Health Prize Administrators (Kaggle)

First congratualations to the 'Market Makers' team, well done!

I would like to respond to some of the questions brought up here.

Well, if you look at the leaderboard you will see that the top Netflix contestants who are competing in this challenge are all in the top. This is not because they have all have a medical background (which they don't) but because they are great data miners. It may be disappointing to some but the meaning of the data is indeed for a large part irrelevant. Understanding what the data means will help in choosing the right features and ways to handle them which is helpful but indeed not required.


I don't know what is the official way to do it but I'll try to answer any questions posted on the forum.


If you like to know more about the techniques used you could start by reading http://en.wikipedia.org/wiki/Stochastic_gradient_descent. Many of the papers published about the Netflix Prize will also be helpful. For each model I included the equation for which the error was minimized. Applying stochastic gradient descent to these formula's is really 'all' you have to do. If you need help calculating the required gradients you could use a free online tool like this: http://library.wolfram.com/webMathematica/Education/WalkD.jsp. For the first model I included the resulting update rules as an example. Also all parameter settings for all models are included in appendix A. That should be sufficient to (approximately) reproduce the result but I realise that 30 days is probably too short to learn the techniques and implement everything.


I send Kaggle the complete source code and executable to exactly reproduce the result. I haven't heard yet whether they succesfully reproduced it but it should not be hard. They should also be able to verify that there is no additional data being used and that the published descriptions match the actual implementation.

You can provide feedback through this forum topic, or by contacting Kaggle through the web site or via email.

The prize winners have provided source code for analysis by the judges.

Competitors have access to the papers, not the source code. If you feel that the papers do not provide enough information to replicate the results, please let us know what you think is missing.

I have reproduced the model generation code in our write up on my blog - it was a mess to copy from the pdf as you got the page numbers and header included. The code is in two parts, the SQL to create the data set and then R to build a model. Hopefully in 2 clicks and 25 minutes you should have a file ready for submision.

One point to mention is when this is all set up, you can essentially build another model utilising a different algorithm by just changing a few lines of the R code.

I am keen to here if anyone gets any joy from this code and if the reported leaderboard scores give you around 0.4635. Interestingly this model would have been about 10th most accurate in Willems blend of 21 models.

The code is here...

http://anotherdataminingblog.blogspot.com/2011/10/code-for-respectable-hhp-model.html

Congrats to both teams!

(Also kudos to B Yang for doing better on the hidden set (apparently) than on the leaderboard set)

I read both papers and haven't read the new stuff that Sali Mali posted - but I will be soon - need to eat something first.

It seems both teams had somewhat close results in their individual models and blending benefit.

Willem gave excellent detail on the scores for each model in his blend - and the final leaderboard score for his blend.

It appears that  both teams best single models did about 0.460345
It appears both teams had about 20 models in their final blend

I'd be curious to know from Willem if he knows or could guess what gain he got from using Ridge Regression vs just plain old linear regression.

I enjoyed reading both papers.

 

 

Thank you so very much. I was able to get public leaderboard score of 0.463549 with this code. Congratulations on the well deserved Heritage Health Prize Round 1 Milestone prize!

I am keen to here if anyone gets any joy from this code and if the reported leaderboard scores give you around 0.4635. Interestingly this model would have been about 10th most accurate in Willems blend of 21 models.

Applying the gradient boosting methods to an otherwise linear regression based model significantly improved my score, so on that alone I appreciate the education (I've got a lot to learn).  Judging from the leaderboards where a few people have shot up in the rankings into the 0.46xxx zone I'm not alone.

Of course I still have like a million questions to get into the 0.45s, but I'll keep working.

---Chip

Hi Willem,

Congrats again on the great work! Could you share more details on how did you blend the 21 models ultilizing the leader board scores, i.e., how to determine the weight of each model. In my view, this is the key to push up the final results. It seems that the blending technique was the key to Netflix competition too. Unfortunately, I did not get chance to participate that great competition. Sample code for the blending will be greatly appreciated. Thanks.

John