Comparing the difficulty of the UKPC from year to year

Information and discussion on Puzzle Competitions

Comparing the difficulty of the UKPC from year to year

Postby dickoon » Mon 25 Jul, 2016 9:08 am

I've been playing around with spreadsheets and the conclusions may or may not be of interest.

There have been six UK Puzzle Championships to date. Here is a graph comparing anonymised solvers' performances on them:

Image

I'm not clear why the labels have been chopped off - is there a maximum width for image links? In any case, from top to bottom, the dark blue line represents 2011, the orange line 2012, the yellow line 2013, the green lien 2014, the brown line 2015 and the light blue line 2016. The symbols - while small - do differ, with those with affected colour vision in mind.

The horizontal axis represents the position of the solver relative to the cohort (the best performing finisher far left, the worst performing finisher who scored at least one point on the right, the median solver midway and so on) and the vertical axis represents the score of the solver, expressed as a percentage of the nominal perfect score assuming no bonus. Accordingly, an all-correct solution with a time bonus earns more than 100%.

I'll be interested to see what conclusions you draw from this, but some starting points might include:

1) The 2016 test was relatively low-scoring in percentage terms. I would additionally note that each year's test has been completely correctly solved by at least one person somewhere in the world, but seldom many at all, which speaks well of the test setting.

2) In general terms, I don't think it's too much of a stretch to identify four different performance levels:
a) the McGowan-Zussman zone (quoted in the order that sounds nicer!) far left,
b) a relatively steep gradient between about 10% cohort and about 30% of the cohort,
c) a relatively shallow gradient between about 30% of the cohort and about 83% of the cohort,
d) a distinct drop-off from about 83% onwards - including those who submitted only one or two answers and then aborted in any particular year.

Following on from this:

Image

This is the same chart as above, but data points from 2011 to 2015 have been marked with a + or a X. Data points marked with + symbols refer to solvers who have participated in the contest in later years. Data points marked with X symbols refer to solvers in their last year of participation. Obviously the participants from 2016 have not been marked either way as it is not clear whether they will participate from 2017 onwards or not.

Again, I'll be interested to see what conclusions you draw from this, but here's what I've got:

1) Every year from 2011-2015, there has been at least one top-six solver who hasn't participated in future years. While you can't make people participate if the date and time don't suit, or if their interest in UKPC puzzles has waned, the potential UK team at the WPC would surely benefit strongly from their participation.

2) If you finish in the bottom 20%, you are no more likely (and, in three of five years, strictly less likely) to participate than not to participate in future years. Speaking as a self-certified, long-established "crap 'un", there have been years where I've been practically (and at least one year where I've been literally) the only bottom-feeder to come back and participate in future years. Now perhaps this would have less of an impact on the potential UK team at the WPC, but it doesn't strike me as an indicator of robust health from metaphorical nose to tail.

3) Maybe I'm not as good with spreadsheets as I'd like, and maybe it's easier with something other than LibreOffice, but it's a bit of a bear to produce a chart which is a pair of scatter graphs overlaid on a line graph.
dickoon
 
Posts: 593
Joined: Mon 21 Jun, 2010 12:52 pm
Location: Stockton-on-Tees, UK

Re: Comparing the difficulty of the UKPC from year to year

Postby GarethMoore » Tue 26 Jul, 2016 12:25 am

Interesting analysis. Putting the UKPC itself aside for a moment, I think the overall difficulty of puzzle competitions has tended to go up with time due to the narrow set of people who keep playing requiring ever tougher puzzles. I think the puzzles tend to be more variants of variants, or crossovers, further disadvantaging new players. The barrier to entry is certainly high, since you need to be familiar with so many different types in order to compete at a certain level. Not that there's necessarily anything wrong with that - but if the aim is to get new people involved, I would certainly say it's important to always have a significant number of puzzles that are approachable to anyone. Similarly, it's important to include puzzles that 'everyone' will be familiar with, so the kind of puzzles you might find in a newspaper.
GarethMoore
 
Posts: 1144
Joined: Thu 24 Jun, 2010 9:27 pm

Re: Comparing the difficulty of the UKPC from year to year

Postby kiwijam » Wed 27 Jul, 2016 4:22 am

I found this year's contest to be harder, or at least to contain more puzzles in the same time-frame, which I guess matches your first chart.
I feel that the time length of 2.5 hours needs further discussion here.
6 years ago I recall there being other contests of this length, now I think there's only the USPC (which I don't enter), and even 2 hours is rare.
The GPs and LMI tend to use 1.5 hours.

I guess Gareth is trying to discover what people enjoy about the UKPC, and what makes them want to enter again.
Would a shorter length be better? Do we want more 'casual' puzzles?
Is it primarily to decide the WPC team, and therefore aims to be hard and varied, or is there value in making it friendlier?
kiwijam
 
Posts: 552
Joined: Wed 01 Jun, 2011 10:03 am

Re: Comparing the difficulty of the UKPC from year to year

Postby berni » Wed 27 Jul, 2016 8:37 am

kiwijam wrote:6 years ago I recall there being other contests of this length, now I think there's only the USPC (which I don't enter), and even 2 hours is rare.
The GPs and LMI tend to use 1.5 hours.


Just a note: The german qualification round (which is held online and open) is always 2.5 hours. But this might be discussable too.
berni
 
Posts: 138
Joined: Mon 01 Nov, 2010 11:38 am

Re: Comparing the difficulty of the UKPC from year to year

Postby detuned » Sun 07 Aug, 2016 9:51 pm

GarethMoore wrote:Interesting analysis. Putting the UKPC itself aside for a moment, I think the overall difficulty of puzzle competitions has tended to go up with time due to the narrow set of people who keep playing requiring ever tougher puzzles. I think the puzzles tend to be more variants of variants, or crossovers, further disadvantaging new players. The barrier to entry is certainly high, since you need to be familiar with so many different types in order to compete at a certain level. Not that there's necessarily anything wrong with that - but if the aim is to get new people involved, I would certainly say it's important to always have a significant number of puzzles that are approachable to anyone. Similarly, it's important to include puzzles that 'everyone' will be familiar with, so the kind of puzzles you might find in a newspaper.


I haven't been on the forums for a while, but I definitely agree with Gareth here. One of the best things about the UKPC in previous years - a feature that IMO made it far better than much of the competition - is that there was a decent variety of entry level or newspaper familiar puzzles. The other great thing is that the novelties were genuinely novel, as opposed to layering on the variants of variants or crossovers that Gareth mentions.

Still, that's an ideal world and the competitions are ultimately based upon voluntary contributions by all concerned
detuned
 
Posts: 1512
Joined: Mon 21 Jun, 2010 2:25 pm
Location: London, UK


Return to Puzzle Competitions

Who is online

Users browsing this forum: No registered users and 1 guest