As far as I can tell, people/teams that continuously submit are just randomizing their input variables slightly and randomly changing weights.  For the life of me I can't figure out how this would help HPN achieve its goal.  As soon as the actual target changes, all top ranking submissions would be useless.

Is this just a game of submit as many random variations as you can and hope you are the closets?

Am I missing something? Why would anyone submit more than a few entries?

They aren't randomizing parameters to hope they will improve their score - it has been shown to improve the MODELS themselves. I read elsewhere in the Netflix competition - if you take two models - that give the same score and have the same mean - it is IMPOSSIBLE for them to be worse than the individual models themselves. It is counterintuitive, but true (not sure about impossible), but every time I have done it - it helps.

Read more about blending, ensembling, and the like and eventually some of it makes sense. There is a good thread on blending in the "predicting a biological response" forum on kaggle right now - with some good references.

The link for the discussion I mentioned - see second post for some references:
http://www.kaggle.com/c/bioresponse/forums/t/1889/question-about-the-process-of-ensemble-learning

thanks for your reply.

So you're saynig that people are rewriting their models daily?

I find that hard to beleive.  Extremely hard to believe.  The current leader produced 294 distinct models?  Really?  Much more likely people are changing weights and inputs.  Which is equivalent to randomizing output.  

At the end of the day, it doesnt matter how fancy your [ f(X)=C]  is... if all your doing is jiggering X a little to get closer to C.  The same logic applies if you are Just doing this [f(x1) + f(x2) ...+ f(xn) = C].  

You randomize your functions, and then you end up with maybe a couple dozen variations, not hundred or thousands.  Unless your playing with X.

In a pedantic sense, a model that differs from another in only a few parameters is still a different model, so, yes. Teams are producing distinct models daily.

As far as helping HPN, it's not really about that. Sure, we'd like to come up with a model that helps them with their goals, but there's also the competition to consider. In my opinion, the model that helps HHP the most, won't necessarily be the top model, because the top model's goal isn't necessarily to be elegant or insightful: it's to minimize RMSLE and win the challenge. This can lead to models that are specific to this competition and are potentially computationally expensive.

>>because the top model's goal isn't necessarily to be elegant or insightful: it's to minimize RMSLE and win the challenge.

yeah, thats what it seems like to me.  I suppose I was under the impression that anyone willing to plunk down $3mm on the table actually wants something useful as a result.  

In any case, time to start guessing

I think there's a very wide spectrum between "rewriting a model daily" and "randomizing output."  We have many submissions that similar to each other.  If we have a model that we like, but we find a few new variables that we think will be predictive of the target, we will re-run the model with the extra explanatory variables.  That's not starting from scratch, but it's not randomly tweaking coefficients either.

That said, the ensembling process described in Chris's link is probably the #1 factor explaining the high submission counts for all the top teams.

If I work on something, at the end of the day I make a submission: why not? It isn't guessing, I knew what I submitted today was not going to do well, but there is no down side.

Really, we often wished we could submit more than one model to see the results of something we were experimenting.

Having only come across the competition recently, I was also surprised at the number of submissions by the teams, and had the same impression as smartersoft. On the other hand, at the end of the day, I, too want to know if I the current approach I am taking is significantly different from my other approaches, so, yes, after making significant changes to my models, I will make a submission. And, since the results of different predictions vary so much between years 2 and 3, it helps to know which model best fits year 4.

Nevertheless, none of any team's submissions so far comes close to the desired result, and shaving a few ten-thousandths off the score every other day with decreasing returns is unlikely to succeed. With that in mind, once I have determined that a given model has little chance of success, I scratch it and move on. Keep in mind, that other teams are using standard methods already. Also, HPN wouldn't be offering this prize if it was easily achieved. Good or bad, my approach is not just to think outside the box, but to throw the box away. Good luck everyone.

0.400000 isn't really a desired result - it is what is needed to get the $3 million. Someone smarter than me posted info before suggesting that that would be a much higher goal than what was commonly though possible with health care data.

This is a competition - I would take every bit I could - no matter how small and submit every day (except for competitive game theory reasons).

We went to the moon using standard methods, slide rules, stop watches and the like. The wheel still works pretty well, and there aren't many flying cars. I think people are expecting a little too much, but that is the great thing about this competition - if someone has a better method - it doesn't matter where you went to school, what race you are, how old you are (within reason), or how accepted your methods are.

I am trying LOTS of really what I consider ingenious methods - the problem is no matter how great I think they are - they aren't beating out the less ingenious methods I use.

It really upsets me when I have such a strong feeling about one of my theories and am proven wrong by the facts.

Believe in the data, you must.

If feedback from Leaderboard Score you did not have, in errant thinking you would persist.

Ignore Yoda, you must.

To the dark side, submit.

If that's what the leaders are doing, it's not going to get them the grand prize - only 30% of the data is used for leaderboards - if they are essentially overfitting the data by doing a parameter optimization on the unknown data, day by day, they are going to fit that 30% very well, but not the 100%.

Now, on the other hand, the data provided for training in this challenge is so atrociously bad that they may actually be using the hidden (presumably less useless / obfuscated) set to train, via the submission mechanism. In that case, they are very clever and should be commended. =)

Good point.  To be clear: this is not a $3M compitition.  This is a $500K competition with a $2.5M bonus if anyone can hit the impossible threshold of 0.40.