There have been several concerns raised in the forum about the impact and interpretation of Rule 13 on the contest, which states that conditional milestone winners must disclose their "Prediction Algorithm and documentation" to the website for competitor review and commentary? In particular, there are unanswered questions with regard to inconsistencies and/or potentially unfair advantages arising from this rule. Can you comment on the following specific items so the community has firm, consistent and realistic expectations as we approach the Milestone 1 date?
- Is it inconsistent, as Sali Mali pointed out in another thread, to require documentation of the winning algorithms be publicly disclosed to all competitors given Rule 20, Entrant Representations? It seems that this disclosure will encourage other competitors to use aspects of the winning Prediction Algorithm which cause violation, directly or otherwise, of (i) - (iii) and possibly (iv) of that Rule.
- Can you clarify that code, libraries and software specifications are *not* required to be publicly disclosed to competitors? These materials and intellectual property appear to be referenced separately from "Prediction Algorithm and documentation."
- Will Kaggle or Heritage have a moderation or appeals process for handling competitor complaints? From the winning entrant's point-of-view, they would not want to be forced through the review process to allow back-door answers to code and libraries which accelerate a competitor's integration of the winning solution.
- Can you comment on the spirit and fairness of the public disclosure of the Prediction Algorithm documentation and it's impact on competitiveness? In particular, if the documentation truly does meet the requirement of enabling a skilled computer science practitioner to reproduce the winning result, then this places the winning team at an unfair disadavantage: all competitors will have access to their algorithms and research, in addition to the winning algorithm.
- Can you provide more detailed clarification on the level of documentation required by conditional milestone winners? The guideline provided by the rules would cover a range of details and description spanning from "lecture notes" to "detailed tutorial" to "whitepaper" to "conference paper", etc.
- Can you comment on the reproducibility requirement? For example, it is possible to construct algorithms with stochastic elements that may not be precisely reproducible, even using the same random seed-- is it sufficient for these algorithms to reproduce the submission approximately? What if they don't reproduce exactly, or reproduce at a prediction accuracy that is worse than the submission score, possibly worse than other competitor submissions?