Iowa received a mediocre “C” grade on the Sunlight Foundation‘s new Open Legislative Data Report Card, designed to compare “how state legislatures make their data publicly available.”
James Turk explained the concept of this report card on the Sunlight Foundation’s blog today. Analysts looked at official legislative websites in every state.
After some consideration, we came up with six criteria on which each state could be evaluated, based on six of the Ten Principles for Opening Up Government Information: completeness, timeliness, ease of access, machine readability, use of commonly owned standards and permanence. We omitted four of the original ten criteria (primacy, non-discrimination, licensing and usage costs) that tended not to present serious differences between states.
Evaluating each state on each criteria was a large task, and with community support we ensured that each state was evaluated by multiple people. After the evaluation was complete, we converted the qualitative data on how a state performed to numeric scores (specific scoring details are available on the report card itself). After summing these scores, states were also assigned a letter grade according to where they fell among their peers. A state with a net score below negative one was given an F, a negative one or zero became a D. With the average total score among states being a 1.5, we gave states with a net score of one or two a C, three became a B, and four and above became an A.
The final breakdown was 8 As, 11 Bs, 20 Cs, 6 Ds, and 6 Fs.
Iowa’s net score of 2 put us in the “C” group. The official legislative website has some good features but could be improved.
This page explains the methodology used to compile each state’s scores.
For “completeness,”
We evaluated each state on the data collected by Open States: bills, legislators, committees, votes and events. We also took note if a state went above and beyond to provide this information and other relevant contextual information such as supporting documents, legislative journals and schedules. Points were deducted for missing data, often roll call votes.
Iowa didn’t lose any points here, as the official legislative website “provides [the] full breadth of legislative artifacts Open States collects: bills, legislators, votes, and committees.” Five states lost a point in this category.
Iowa did well in the “timeliness” category as one of 27 states that received one point: the legislative website is updated multiple times a day. Some states received zero points, meaning that site updates happen “once or twice daily, typically at the end of the legislative day,” or negative one point, if updates ” take longer than 24 hours to appear on the site, often up to a week.”
For “ease of access”,
Common web technologies such as Flash or JavaScript can cause problems when reviewing legislative data. We found that the majority of sites work fairly well without JavaScript, but some received lower scores due to being extremely difficult to navigate, impossible to bookmark bills, and in extreme cases, completely unusable.
The best state legislative websites received one point for being “exceptionally well layed out by multiple evaluators, no issues with Javascript.” Iowa was one of the 36 states that received zero points: “Site was deemed average by those that evaluated it and/or had minor Javascript dependencies.” The worst legislative websites lost one or two points in this category.
The worst category for Iowa was “machine readability”:
For many sites, the Open States team wrote scrapers to collect legislative information from the website code-a slow, tedious and error prone process. We collected data faster and more reliably when data was provided in a machine-readable format such as XML, JSON, CSV or via bulk downloads. If a state posted PDF image files or scanned documents, it received the lowest score possible.
2 Essentially all data can be found in machine-readable formats.
1 Lots of data in machine readable format but substantial portions that still required scraping HTML.
0 No machine readable data but standard screen scraping techniques applied.
-1 Site had information that was much more difficult than average to collect. (Data only accessible via PDF or that required screen scraper to emulate Javascript.)
-2 Site had information that was unaccessible to Open States due to use of scanned PDFs.
Iowa was one of 13 states in the “minus one” group here.
For “use of commonly owned standards,” Iowa was one of 48 states to receive a zero, meaning that the legislative website “provided bills in PDF and/or HTML format and nothing better (plaintext, ODT, etc.).”
Iowa’s best score was in the last category, “permanence.”
Many states move or remove information when a new session starts, much to the dismay of citizens seeking information on old proposals and researchers that may have cited a link (e.g. http://somelegislature.gov/HB1 vs http://somelegislature.gov/201… only to see it point to a different bill in the following session. Tim Berners-Lee, inventor of the World Wide Web, wrote an article declaring Cool URIs Don’t Change and we agree.
This poses a particular challenge to us since every page on OpenStates.org points to the page we collected data from, but if a state changes their site then users lose the ability to check us against the original source. Most (but not all) states are good about at least preserving bill information, but few were equally as good about preserving information about out-of-office legislators and historical committees, equally important parts of the legislative process.
2 All information is available in a permanent location and data goes back a reasonable amount of time (a decade or so).
1 Almost all information has a permanent location but a single data set doesn’t. (Or a recent change to the site has wiped out historical links but information appears to be preservable going forward.)
0 Legislator & committee information lacks a permanent location (such as committees and legislators) but most is acceptable.
-1 Ability to link to old information is badly damaged and and/or there is less than a decade of historical information.
-2 Vital information like bills or versions lack a permanent location.
Iowa scored the best possible score of two points here, as did 33 other states. I just used this feature of the Iowa legislative site yesterday in order to look up the 2011 co-sponsors of the constitutional amendment on marriage.
Here’s hoping the webmasters who work hard on Iowa’s site will steal some good ideas from the best state legislative websites around the country: Arkansas, Connecticut, Georgia, Kansas, New Hampshire, North Carolina, Texas, and Washington.