Challenges in Machine Learning have proven to be efficient and cost-effective ways to quickly bring to industry solutions that may have been confined to research. In addition, the playful nature of challenges naturally attracts students, making challenge a great teaching resource. Challenge participants range from undergraduate students to retirees, joining forces in a rewarding environment allowing them to learn, perform research, and demonstrate excellence. Therefore challenges can be used as a means of directing research, advancing the state-of-the-art or venturing in completely new domains.
Because challenges have become stream line in the execution of Machine Learning projects, it has become increasingly important to regularly bring together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. Following the success of last year's workshop (
http://ciml.chalearn.org/), in which a fruitful exchange led to many innovations, we propose to reconvene and discuss the new avenues that have been explored and lay the basis for further developments. We are particularly interested in following progresses made in two conceptually important directions:
1) Open innovation: Organization of contests in which data are made available and the contestants must both formalize and solve a problem (with some constraints), leaving more freedom to creativity, while giving more difficulty to the organizers to objectively assess the results.
2) Coopetitions: Organization of contests encouraging both collaboration and competition, in an effort to make possible the contributions of many towards a the grand goal of solving the overall problem; this poses to the organizers the problem of rewarding partial contributions.
We also want to closely follow more technical, albeit important aspects:
3) Platforms: New developments including "code submission" (platforms and protocols permitting code submission, as opposed to result submission, allowing fairer standardized comparisons in terms of hardware utilization and easier reproducibility) and " worksheets" or "scripts" facilitating code sharing.
4) Sharing, dissemination, and recognition: Facilitate sharing resources, including data, means of data collection and annotation, challenge announcements, best practices, challenge templates, publication channels, etc.; creation of awards to recognize academic services rendered by the various actors of challenge organization.
Morning session (9:00 am-12:00 pm)
9:00 - Welcome and introduction. Evelyne Viegas (10 min)
9:10 - Invited talk, Challenges in Medical Image Analysis: Comparison, Competition, Collaboration, Bram van Ginneken (40 min)
9:50 - Break (30 min)
10:20 - Invited talk, Techniques and Technologies for Efficient and Realistic Benchmarks: Examples from the MediaEval Multimedia Benchmark and CLEF NewsREEL, Martha Larson (40 min)
11:00 - Discussion: Open Innovation, Balazs Kegl and Ben Hamner moderators (60 min)
12:00 - Break (30 min)
Break-out session on AutoML challenge (12:00 - 14:30). Free lunch will be served for the participants.
12:30 - Presentation of the AutoML challenge. Isabelle Guyon (30 min) -- Announcement of the new GPU track.
13:00 - Automated Machine Learning: Successes & Challenges. Frank Hutter. Team aaad_freiburg. First place AutoML1 phase, second place AutoML2 phase. (30 min)
13:30 - Sensible allocation of computation for ensemble construction. James Lloyd. Team jrl44/backstreet.bayes. First place AutoML2 phase, second place AutoML1 phase.
14:00 - Scalable ensemble learning with stochastic feature boosting. Eugene Tuv. Team ideal.intel.analytics. First place Final0 phase, second place Final1 phase.
14:30 - Break (30 min)
Afternoon session (15:00-18:30pm)
15:00 - Invited talk, Lessons Learned from the PASCAL VOC Challenges, and Improving the Data Analytics Process, Chris Williams (40 min)
15:40 - Discussion: Coopetitions, Evelyne Viegas and Isabelle Guyon moderator (60 min)
16:40 - Break (20 min)
17:00 - Contributed talk, Academic Torrents: Scalable Data Distribution, Henry Z. Lo and Joseph Paul Cohen (30 min)
17:30 - Open discussion.
18:30 - Adjourn
http://ciml.chalearn.org/
