Success Rate Methodology
One comparison that is useful to know is the percentage of requests through WhatDoTheyKnow were successful (met in full) compared to the success rate in the Cabinet Office release. This helps us determine how typical the requests processed through WhatDoTheyKnow are.
Because the 'success' of a request in WDTK is self-reported, working this out presents a problem. There are quite a sizeable amounts of WDTK requests that are waiting to have their statuses updated.
Including these all as 'unsuccessful' gives a harsh view of WDTK's success rate - in 2016 including makes our success rate 12.8% worse than the official numbers. But if you take them out they’re practically the same as the official numbers with WDTK being half a percentage point more successful.
But while the second number feels nicer because it’s closer, is it actually more meaningful? If people whose requests failed are less likely to tell us about the result, then removing all these numbers will make WDTK’s success rate seem better because we’re removing failures. Comparing over time is also difficult because there will be a different composition to entries still marked as waiting_reply in 2015 than 2016 (where there is still the possibility of update). To know what’s going on we have to evaluate exactly how the pool of uncategorised replies breaks down.
We built a process to auto-categorize these based on content on the reply text and the attachments. There are lots of possible hints a request has failed (“refusing your request”, “is not held”, “we do not hold records”, etc) but in general there was less boilerplate language when a request was successful (not none, just less consistent). It is possible to pick these up by the type of attachment in some cases (excel files are a good signifier of success) but it’s an error prone process.
However, as we really only want to know the success rates across the database, we just have to understand the aggregate error of this process and adjust. The automated conversion turns 75,232 ‘waiting for response’ requests into :
| Status | Count |
|---|---|
| NO_FURTHER_REPLY | 4907 |
| SUCCESS | 20,770 |
| AWAITING_RESPONSE | 24,667 |
| REFUSED | 4,364 |
| HOME_OFFICE_ACKNOWLEDGED | 2,66 |
| NEVER_ACKNOWLEDGED | 12,453 |
| PARTIALLY_SUCCESSFUL | 7,787 |
| NEEDS_CLARIFICATION | 18 |
Then for each of these categories, we reviewed a random 100 FOI requests from 2016 to see how many the automated process had got wrong. Folding those categories back down to only four (some such as NEVER_ACKNOWLEDGED are 100% correct by default) – Our automated process had the following accuracy rates:
| successful | partially_succesful | waiting_clarification | waiting_response | accuracy | |
|---|---|---|---|---|---|
| successful | 85 | 2 | 0 | 0 | 85% |
| refused | 16 | 11 | 4 | 1 | 68% |
| partially_successful | 17 | 30 | 1 | 0 | 30% |
| awaiting_response | 31 | 2 | 16 | 25 | 25% |
Based on this we adjusted the counting process so that that one request automatically assigned as successful was actually counted as 0.85 successful, 0.13 unsuccessful and so on. The idea being across the whole dataset this should roughly give us a correct allocation. The correction is most true for 2016 (which is the numbers I care most about) – and might be drifting from accuracy in earlier years as the breakdown in reasons why the automatic process might be different (and hence correction rates correspondingly might also are wrong).
So all our ‘waiting for replies’ now break down like this by year:
| all data | successful | partially_successful | waiting_clarification | waiting_response | unsuccessful |
|---|---|---|---|---|---|
| 2010 | 19% | 2% | 6% | 9% | 64% |
| 2011 | 19% | 2% | 5% | 8% | 66% |
| 2012 | 27% | 3% | 5% | 7% | 58% |
| 2013 | 37% | 5% | 6% | 9% | 43% |
| 2014 | 38% | 5% | 6% | 10% | 41% |
| 2015 | 38% | 5% | 6% | 9% | 42% |
| 2016 | 41% | 6% | 5% | 8% | 39% |
| Grand Total | 36% | 5% | 6% | 9% | 45% |
Which feels about what we’d expect – awaiting replies are mostly unsuccessful, but there are still a lot of successful ones in there. When we focus in on just those covered by the Cabinet Office release (central government departments and agencies), we get this:
| central gov data | successful | partially_successful | waiting_clarification | waiting_response | unsuccessful |
|---|---|---|---|---|---|
| 2010 | 19% | 2% | 6% | 10% | 63% |
| 2011 | 19% | 3% | 5% | 8% | 65% |
| 2012 | 35% | 7% | 4% | 6% | 49% |
| 2013 | 39% | 7% | 5% | 7% | 41% |
| 2014 | 39% | 8% | 4% | 14% | 34% |
| 2015 | 40% | 8% | 4% | 10% | 39% |
| 2016 | 41% | 7% | 4% | 9% | 38% |
| Grand Total | 36% | 5% | 6% | 9% | 45% |
In 2014, 2015, 2016 the central government unupdated requests actually break down slightly in favour of successful requests – which explains how our figures show a marginally better success rate for WDTK when the adjusted numbers are included.
When this adjusted methodology is applied to the database, we now have a success rate of 38.4% in 2016 which compares favourable to the 34.22% of official statistics. Given the process taken to get there, it'd be a mistake to read too far into this. What it does suggest is that WDTK requests are representative in terms of success across central government to all requests in total. While we can't break this down on a department level without checking the adjustments, it suggests conclusions drawn from the corpus of WDTK data might be applicable to FOI requests in general. That success rates in WDTK for non-central government public bodies are far higher might similarly be suggestive that this is generally true - but confirmation of this will require further research.
Similarly to make more comprehensive conclusion this exercise should be repeated for several different years and use multiple coders to ensure results are reproducible.