I was wondering whether it would be possible to create a submission API so that we could submit from software?

It's fairly common for my team to pre-generate a few solutions then take a break. But we still have to log in every day to submit them. If there was an API we could just set up a cron job that uploads the solutions and e-mails us the result.

Failing that, could you provide the ability to upload multiple solutions and auto-submit one every 24 hours?

Queued submissions are an interesting idea.

What if you could fill your submission queue with many submissions (i.e. 2-3 weeks worth) and then at the end of each day if you hadn't used up your submission quota and had X entries left for the day (1 entry in the case of this competition), the next X entries in your queue would become active. It'd be just as if you submitted the entry at the end of the day. If we did it this way, you could still submit during the day and choose to use that submission immediately rather than wait in the the queue.

Alternatively, we could make it such that the concept of submissions is always first-in first-out (FIFO). Thus, if you upload a submission but already had submissions in the queue then your submission would go to the back of the queue. In this approach, the next queued submission would go live at midnight each day (and thus use up your submission for the entire day). This approach is slightly less flexible, but is a bit simpler to understand (and implement :)).

If I take the simpler FIFO approach, would that work for you?

I think a submission queue (either pure FIFO queue, or something that users can reorder) would be a great feature.

Interesting idea Jeff. I'm not sure whether I like it or not... I can see the convenience of it, but I also wonder it might lead to less good results. Forcing people to step back and think for 24 hours before submitting a new answer can be quite useful; maybe others are more disiplined than me, but I can recall many times that this forced me to think more laterally and deeply.

What do others think about this proposed feature?

like netflix :)

hmmm - I am torn on this. 

I almost didn't get in my daily sub yesterday due to a power outage :) ... but,

I don't think I like the idea of people just queuing up 100 submissions, but perhaps a middle ground - such as being able to have three to five in a queue - and if midnight comes with no sub - you can trigger the backups.

However - I don't have a very strong feeling one way or another - I guess I would like a SMALL queue if I were to choose.

I tend to agree, but see some merit to having a backup for when your day just doesn't go well :)

Normally I submit day-by-day because typically I generate a solution based on the previous result. But occasionally I'll be away (or want to be away) from the computer and want to pre-schedule some entries.
As I said earlier, my preference is for a submission API. It would also simplify regular submissions since I could automate some other steps that are currently done manually.
But if we're going the pre-scheduled route, I'd prefer a calendar where we could upload submissions for particular days - and replace or delete them as necessary. A straight up queue is a bit of a hassle to coordinate between multiple team members.

One other thing - if we can pre-schedule submissions, my preference is to be notified of the submission result by e-mail.
I guess a simple way to implement pre-scheduling (from a UI perspective) is to add a calendar to the "Make a submission" page. If a date is selected, the submission is scheduled for that date. And the "Submissions" page could show future as well as past entries.
It might make sense to schedule the time as well - some teams might want to submit ASAP so they can make use of the result, others may want to delay until the last moment in case they come up with something better.

Even a simple queue feature would be useful. Minimally, you need a queue and a way to delete pending (queued) submissions. Everything else is "nice to have".

I think submission queue will be a useful feature, there're times when you'll be away for days but don't want to miss the submission slots.

While we're on this subject, may I suggest (again) increasing the number of submissions per day based on the size of the team ? You can cap it at maximum of 4 or 5 submissions per day.

One submission per day is not a big deal for small teams, but if your team is bigger than a few people, it will be hurting you.

On second thought, the real issue here is your daily submission slot is gone if you don't use it. So perhaps a better solution is to allow unused submission slots to accumulate with a maximum limit (of 10 maybe).

Absolutely untrue: submission limits are handicaps for everyone, regardless of team size.  It would be grossly unfair to let larger teams have more submissions!

Personally, I think there is not much need for a submission queue, though it might help some teams.  What I think would really be beneficial is to change the current system to use a tickets metaphor.  Each team gets a fixed number of submission tickets for each milestone.  When you use them is up to you.  This maintains fair handicapping, but allows teams to match their submissions to their workflow.

As a bonus, such a system naturally lends itself to monetization for Kaggle.  For a small fee, any team could buy a bonus pack of submission tickets.  Obviously the fee should be small, and the maximum number of ticket add-ons should be limited, in the interest of fairness.  Such a system maintains the handicap -- any team can acquire the same total number of submission tickets as any other team -- and allows extra submissions for those that feel they need them.

Andy

Please don't change the rules. Period. This would be unfair to hard working submitters who are regularly submitting and enjoying the thrill of waiting for results.

1 Attachment —

• Untitled.png (20.43 KB)

I support no rule changes until Milestone boundaries.

I like the idea of a short FIFO queue, with the option of deleting queued entries. If the end of the day comes and I have not made a submission, the next one in the queue goes.

Maybe implement this after the milestone?

I think B Yang has the best idea.  Just allow people to back submit.

Having just submitted a 0.5138 score hopefully due to some bug I haven't tracked down yet, I'm voting in favor of back submittal. :-)

I don't think more than a few are needed.. 2 or 3 maybe?

On the other hand, this will upset the balance that is causing the conversation in this thread: http://www.heritagehealthprize.com/c/hhp/forums/t/815/hidden-rules-for-team-mergers/5338#post5338

Namely that, when merging teams, the combined team can't have more submissions than they would have been allowed if they were one team the whole time.  Queuing submissions or allowing back submissions is going to result in higher submission rates and will further limit the ability of teams to merge later.

i prefer back submissions, or what i immediately thought of as rollover submissions :), over queuing. i actually think it is a great idea. similar to wireless companies that truncate your minutes when you change your plan, the total number of back submissions accrued could be truncated to a limit when teams merge. that would avoid gaming the system by getting non-contributors to join your team to build up submissions. those who submit daily will continue to have the advantage because they will get feedback sooner, however those who cannot submit daily for whatever reason will not be penalized additionally by missing submissions. this setup retains the benefit highlighted by jeremy as well.

The idea of back submissions is nice but when implemented I will almost stop doing any submissions at all until after milestone 3 just to have a lot available when they are most useful. In my opinion making submissions early is not an advantage. A new comer who starts just now after milestone 1 will find a lot of useful information in the milestone 1 papers and can make better submissions now then I could when I started 6 months ago. Therefore I think it is best to view each milestone as a separate competition in terms of submissions. Within such a 6 month period between prizes having back submissions would be great. After a milestone all counters could be reset giving everyone equal chances for the next prize.

That is a very good point and makes total sense to me.

hi, newcomer here. i was wondering what is the rationale for throttling submissions to once a day? is it simply a matter of server resources? or is there some other reason for this?

@Lucidrains, If I had the ability to do infinite submissions, I could hit the server hundreds of times a day with semi-random submissions until I hit on a solution with a great score.

yeah, i understand, but once a day? i'm looking at the number of teams, and it just seems like for only 500 players, a well optimized server shouldn't even sweat if submissions were increased 10x a day.

this is just my perspective as a server administrator and website owner...

@lucidrains, You may be misunderstanding me.  The issue isn't one of server load.  The issue is that I could use a high number of submissions as a modeling technique.

yeah no i get that for sure. i just think once a day is a tad bit too infrequent. but that's just me.

 

thanks for the prompt response!

Any news regarding this issue? I strongly support the idea of having a submission queue!

I propose a very simple submission queue of 10 days (10 submissions) for the very simple reason of going on vacation and not wanting to lose submission chances.  Also the queue should be editable so that you can change your 10 submissions if your technique radically changes in the meantime.

While I don't think it's necessary to build an API, I think allowing this feature on the webpage would be really helpful (and less sadistic on Kaggle's part... had to log during memorial day weekend....) while not really affecting the nature of what is currently going on in the competition.

thanks

 

 

 

 

I think even if we didn't have an automatic queue - even having a little side cache of files on kaggle that you could then manually submit would be nice. That way I could use my smart phone when out and choose it instead of driving all the way back to my place to make a submission.

An API would be great.  Is there any chance of this (we can implement our own queues and manage them ourselves!)?

 

Thanks

We're attracted to the idea of having a 'bank' of extra submissions (besides the daily limit). Among other reasons, this would definitely be nice for people joining the contest late. I think it would be small, maybe 10 or so?

An API or a bank of submissions are also nice ideas -- not sure when we'd get to them, but I like it. But a small bank of submissions might be a smaller/easy quick improvement. (Yeah, I realize they're really two separate issues.)