<12>
mkwan's image
Rank 17th
Posts 11
Thanks 1
Joined 5 Apr '11
Email User

I was wondering whether it would be possible to create a submission API so that we could submit from software?

It's fairly common for my team to pre-generate a few solutions then take a break. But we still have to log in every day to submit them. If there was an API we could just set up a cron job that uploads the solutions and e-mails us the result.

Failing that, could you provide the ability to upload multiple solutions and auto-submit one every 24 hours?

Thanked by Zach
 
Jeff Moser's image
Jeff Moser
Kaggle Admin
Posts 404
Thanks 215
Joined 21 Aug '10
Email User
From Kaggle

Queued submissions are an interesting idea.

What if you could fill your submission queue with many submissions (i.e. 2-3 weeks worth) and then at the end of each day if you hadn't used up your submission quota and had X entries left for the day (1 entry in the case of this competition), the next X entries in your queue would become active. It'd be just as if you submitted the entry at the end of the day. If we did it this way, you could still submit during the day and choose to use that submission immediately rather than wait in the the queue.

Alternatively, we could make it such that the concept of submissions is always first-in first-out (FIFO). Thus, if you upload a submission but already had submissions in the queue then your submission would go to the back of the queue. In this approach, the next queued submission would go live at midnight each day (and thus use up your submission for the entire day). This approach is slightly less flexible, but is a bit simpler to understand (and implement :)).

If I take the simpler FIFO approach, would that work for you?

Thanked by Zach
 
DanB's image
DanB
Kaggle Admin
Rank 2nd
Posts 67
Thanks 48
Joined 6 Apr '11
Email User
From Kaggle

I think a submission queue (either pure FIFO queue, or something that users can reorder) would be a great feature.

 
Jeremy Howard (Kaggle)'s image
Posts 167
Thanks 62
Joined 13 Oct '10
Email User

Interesting idea Jeff. I'm not sure whether I like it or not... I can see the convenience of it, but I also wonder it might lead to less good results. Forcing people to step back and think for 24 hours before submitting a new answer can be quite useful; maybe others are more disiplined than me, but I can recall many times that this forced me to think more laterally and deeply.

What do others think about this proposed feature?

 
Chris Raimondi's image
Rank 20th
Posts 194
Thanks 90
Joined 9 Jul '10
Email User

either pure FIFO queue, or something that users can reorder


like netflix :)

hmmm - I am torn on this. 

I almost didn't get in my daily sub yesterday due to a power outage :) ... but,

I don't think I like the idea of people just queuing up 100 submissions, but perhaps a middle ground - such as being able to have three to five in a queue - and if midnight comes with no sub - you can trigger the backups.

However - I don't have a very strong feeling one way or another - I guess I would like a SMALL queue if I were to choose.

but I also wonder it might lead to less good results

I tend to agree, but see some merit to having a backup for when your day just doesn't go well :)

 
mkwan's image
Rank 17th
Posts 11
Thanks 1
Joined 5 Apr '11
Email User

Normally I submit day-by-day because typically I generate a solution based on the previous result. But occasionally I'll be away (or want to be away) from the computer and want to pre-schedule some entries.
As I said earlier, my preference is for a submission API. It would also simplify regular submissions since I could automate some other steps that are currently done manually.
But if we're going the pre-scheduled route, I'd prefer a calendar where we could upload submissions for particular days - and replace or delete them as necessary. A straight up queue is a bit of a hassle to coordinate between multiple team members.

 
mkwan's image
Rank 17th
Posts 11
Thanks 1
Joined 5 Apr '11
Email User

One other thing - if we can pre-schedule submissions, my preference is to be notified of the submission result by e-mail.
I guess a simple way to implement pre-scheduling (from a UI perspective) is to add a calendar to the "Make a submission" page. If a date is selected, the submission is scheduled for that date. And the "Submissions" page could show future as well as past entries.
It might make sense to schedule the time as well - some teams might want to submit ASAP so they can make use of the result, others may want to delay until the last moment in case they come up with something better.

 
JJJ's image
JJJ
Posts 51
Thanks 10
Joined 9 Apr '11
Email User

Even a simple queue feature would be useful. Minimally, you need a queue and a way to delete pending (queued) submissions. Everything else is "nice to have".

 
B Yang's image
Rank 2nd
Posts 255
Thanks 71
Joined 12 Nov '10
Email User

I think submission queue will be a useful feature, there're times when you'll be away for days but don't want to miss the submission slots.

While we're on this subject, may I suggest (again) increasing the number of submissions per day based on the size of the team ? You can cap it at maximum of 4 or 5 submissions per day.

One submission per day is not a big deal for small teams, but if your team is bigger than a few people, it will be hurting you.

 
B Yang's image
Rank 2nd
Posts 255
Thanks 71
Joined 12 Nov '10
Email User

On second thought, the real issue here is your daily submission slot is gone if you don't use it. So perhaps a better solution is to allow unused submission slots to accumulate with a maximum limit (of 10 maybe).

 
andywocky's image
Posts 18
Thanks 8
Joined 17 Jun '11
Email User

B Yang wrote:

While we're on this subject, may I suggest (again) increasing the number of submissions per day based on the size of the team ? You can cap it at maximum of 4 or 5 submissions per day.

One submission per day is not a big deal for small teams, but if your team is bigger than a few people, it will be hurting you.

Absolutely untrue: submission limits are handicaps for everyone, regardless of team size.  It would be grossly unfair to let larger teams have more submissions!

Personally, I think there is not much need for a submission queue, though it might help some teams.  What I think would really be beneficial is to change the current system to use a tickets metaphor.  Each team gets a fixed number of submission tickets for each milestone.  When you use them is up to you.  This maintains fair handicapping, but allows teams to match their submissions to their workflow.

As a bonus, such a system naturally lends itself to monetization for Kaggle.  For a small fee, any team could buy a bonus pack of submission tickets.  Obviously the fee should be small, and the maximum number of ticket add-ons should be limited, in the interest of fairness.  Such a system maintains the handicap -- any team can acquire the same total number of submission tickets as any other team -- and allows extra submissions for those that feel they need them.

Andy

 
Sarkis's image
Posts 41
Thanks 5
Joined 5 Apr '11
Email User

Please don't change the rules. Period. This would be unfair to hard working submitters who are regularly submitting and enjoying the thrill of waiting for results.

 
andywocky's image
Posts 18
Thanks 8
Joined 17 Jun '11
Email User

I support no rule changes until Milestone boundaries.

Thanked by Sarkis
 
Zach's image
Rank 9th
Posts 362
Thanks 96
Joined 2 Mar '11
Email User

I like the idea of a short FIFO queue, with the option of deleting queued entries. If the end of the day comes and I have not made a submission, the next one in the queue goes.

Maybe implement this after the milestone?

 
JJJ's image
JJJ
Posts 51
Thanks 10
Joined 9 Apr '11
Email User

B Yang wrote:

On second thought, the real issue here is your daily submission slot is gone if you don't use it. So perhaps a better solution is to allow unused submission slots to accumulate with a maximum limit (of 10 maybe).

I think B Yang has the best idea.  Just allow people to back submit.

 
ChipMonkey's image
Rank 63rd
Posts 83
Thanks 22
Joined 20 Mar '11
Email User

Having just submitted a 0.5138 score hopefully due to some bug I haven't tracked down yet, I'm voting in favor of back submittal. :-)

I don't think more than a few are needed.. 2 or 3 maybe?

On the other hand, this will upset the balance that is causing the conversation in this thread: http://www.heritagehealthprize.com/c/hhp/forums/t/815/hidden-rules-for-team-mergers/5338#post5338

Namely that, when merging teams, the combined team can't have more submissions than they would have been allowed if they were one team the whole time.  Queuing submissions or allowing back submissions is going to result in higher submission rates and will further limit the ability of teams to merge later.

 
Mark Waddle's image
Posts 32
Thanks 8
Joined 28 Mar '11
Email User

i prefer back submissions, or what i immediately thought of as rollover submissions :), over queuing. i actually think it is a great idea. similar to wireless companies that truncate your minutes when you change your plan, the total number of back submissions accrued could be truncated to a limit when teams merge. that would avoid gaming the system by getting non-contributors to join your team to build up submissions. those who submit daily will continue to have the advantage because they will get feedback sooner, however those who cannot submit daily for whatever reason will not be penalized additionally by missing submissions. this setup retains the benefit highlighted by jeremy as well.

 
Willem Mestrom's image
Rank 1st
Posts 34
Thanks 11
Joined 28 Feb '11
Email User

The idea of back submissions is nice but when implemented I will almost stop doing any submissions at all until after milestone 3 just to have a lot available when they are most useful. In my opinion making submissions early is not an advantage. A new comer who starts just now after milestone 1 will find a lot of useful information in the milestone 1 papers and can make better submissions now then I could when I started 6 months ago. Therefore I think it is best to view each milestone as a separate competition in terms of submissions. Within such a 6 month period between prizes having back submissions would be great. After a milestone all counters could be reset giving everyone equal chances for the next prize.

Thanked by Mark Waddle and Signipinnis
 
Mark Waddle's image
Posts 32
Thanks 8
Joined 28 Mar '11
Email User

Willem Mestrom wrote:

After a milestone all counters could be reset giving everyone equal chances for the next prize.

That is a very good point and makes total sense to me.

 
lucidrains's image
Posts 3
Joined 8 Jun '11
Email User

hi, newcomer here. i was wondering what is the rationale for throttling submissions to once a day? is it simply a matter of server resources? or is there some other reason for this?

 
<12>

Reply

Flag alert Flagging is a way of notifying administrators that this message contents inappropriate or abusive content. Are you sure this forum post qualifies?