The 629-page text (.pdf) of the 2008 Farm Bill is so complex and unreadable that the U.S. food policy community has been on the edge of our seats waiting for the USDA/ERS side-by-side comparison unveiled today.

The ERS side-by-side tool compares the new Farm Bill with current law, title by title, so we can finally begin to understand what the law really means.

ERS is the USDA’s Economic Research Service. Their side-by-side webpage, which I think was just published this week, shows the provisions of the previous and the current bill side-by-side. (It’s not a comparison of the bill text, but of summaries of the provisions.)

This is interesting on a number of accounts. First, the fact that it is the USDA making this comparison suggests that everyone agrees that the bill itself is effectively incomprehensible even to professionals and scholars on account of its size and summarizing it is costly enough that only the government would do it, taking three months to prepare.

Second, if this is what was needed to understand the Farm Bill, was it passed without anyone understanding it?

Third- This comparison was made by and for professionals and scholars, not by tech geeks. Why aren’t we talking to them?

The ERS tool comes complete with a seemingly unintentionally hilarious intro video — overly dramatic with background music fit for the Miss Universe competition. (Wilde likened it to “a documentary by Kenneth Burns or an account of a manned mission to the moon”.)

The database is great itself, but the cooler website is MyPyramid Menu Planner (mypyramidtracker.gov) (also out of the USDA). You can enter a typical daily roster of what you eat (with a nice sound effect) and it will tell you how it stacks up for a recommended diet for your age (or for me, how to gain weight to a recommended amount for my age). It feels a little over-simplified, but the simplicity keeps me on the site. I find, not surprisingly, that I probably eat about half of the recommended calories and clearly not enough grain or fruit. Well, I knew this in the abstract, but quantifying it helps direct me to fixing the problem.

I’m sure there are other websites that do similar things, but it’s nice to find a case where the government has both published a comprehensive (well structured, well documented) database and has also built a really nice interface for the data. And on a topic that is really very important to daily life, too.

And with that, I think I will take the rest of the weekend off from civics!

The page updated is here.

The updates were a combination of suggestions from Scott Horvath and Jeremy Fee at the USGS, Kol Peterson from EPA, and me, and really big thanks go to Scott and Kol for reaching out to others for input on Monday and getting the feedback back to Bev Godwin at GSA who runs webcontent.gov who published the changes only a few days later. Scott also notes that additional suggestions could still be considered (his email address is at the bottom of that page).

In making my suggestions, I turned to the Open Government Data Principles and tried to squeeze in as much as I could without overloading the document, and I drew from ideas that came up in the preparation of the Open House Project report. Some of the changes made were:

• It now provides examples of data as being documents, audio/visual recordings, and databases.
• It now says to support “the widest practical range of public uses of
  the data”. It had formerly suggested supporting the “intended” use of
  the website by visitors.
• It notes the benefit of providing data: “New uses of your agency’s
  data may become a valuable public resource that would be out of the
  scope of your own website, such as helping to keep the public informed
  about the work of your agency and supporting civic education and
  participation.”
• There is a new paragraph that I might be misunderstanding but which
  seems to make a suggestion along the lines of the recent “Invisible
  Hand” paper about the agency’s website getting the data the same way the
  public does: “Providing a uniform method to access raw data can also be
  the first step in internal development, accomplishing both goals at
  once. When a uniform method to access data is available, developers and
  web–services can focus on data presentation.”
• It notes that the availability of bulk downloads of data is something
  to consider when building data access.
• It notes some disadvantages of using proprietary formats.
• It recommends that if a proprietary format is needed, a
  non-proprietary format should be used in addition.
• It adds a benchmark to test for success: “One benchmark for
  determining whether data is made sufficiently available is whether the
  public has all of the data needed to replicate any searching, sorting,
  and display functionality provided on the agency’s own website.”
• It notes that consulting the public in the development of data access
  seems to be entailed from OMB policy: “When choosing data formats and
  distribution methods, keep in mind that your agency’s visitors are the
  best judges of their own needs. Agencies must “establish and maintain
  communications with members of the public and with State and local
  governments to ensure your agency creates information dissemination
  products meeting their respective needs” (OMB Policies for Federal
  Public Websites #4A).”

We have a real success story here.

“The new administration should specify that the federal government’s primary objective as an online publisher is to provide data that is easy for others to reuse, rather than to help citizens use the data in one particular way or another.”

And they suggest an interesting way to push that forward:

“The policy route to realizing this principle is to require that federal government websites retrieve the underlying data using the same infrastructure that they have made available to the public. Such a rule incentivizes government bodies to keep this infrastructure in good working order, and ensures that private parties will have no less an opportunity to use public data than the government itself does. The rule prevents the situation, sadly typical of government websites today, in which governmental interest in presenting data in a particular fashion distracts from, and thereby impedes, the provision of data to users for their own purposes.”

I think this is a worthwhile addition to the opengovdata and publicmarkup.org policy documents — if not as a direct recommendation (because I think it may be too much to ask for in a grand form) then noted as a long-term goal or (in terms of the second paragraph I quoted) as a benchmark, a concrete way to tell whether data is open.

The full citation is: Robinson, David, Yu, Harlan, Zeller, William P and Felten, Edward W, “Government Data and the Invisible Hand” (2008). Yale Journal of Law & Technology, Vol. 11, 2008

The GAO is responding to a congressional request for information on electronic documents dissemination, and ends up discussing the GPO, LOC, and FDLP in great detail. Apparently they considered transferring the FDLP from the GPO to the LOC.

Also, the report lists the essential documents which must continue to be printed, regardless of how digitized we become.

The committee rules, as they stand, contradict the new requirements of S1, specifically section 513, which requires public committee proceedings to be posted online within 21 days of the hearing.  I expect that other committees will have to deal with this issue, and the Finance Committee should be applauded for taking the provisions of S1 seriously, and recognizing that their rules will need to be updated to accommodate its requirements.

Committees, as they adapt to new expectations for online information access, should also recognize that these stipulations are only a (very necessary) first step.  Meaningfully access to committee proceedings is only possible through real-time disclosure and digital records management.  This would enable citizens to follow along with hearings that pertain to their interests or expertise as they happen, and also give members of Congress and their staff new tools to help them do their jobs more effectively.  (Multiple committee hearings, floor votes, interviews, staff meetings and who knows what all happen at the same time, the least we can do is make sure members of Congress can find out what happens in the meetings of the committees on which they serve.)

This disclosure, as outlined in the Open House Project report (committee section), must first be timely.  Committee staff have expressed real concerns about posting official transcripts in time, and one solution to that concern may be to post unofficial versions of transcripts first.  In any case, making public access a priority should enable best practices to quickly emerge, and I’m confident in the committees ability to post proceedings quickly.  Senator Salazar was confident of this fact as well, as he remarked when introducing the amendment to the Senate bill: “I should also add that the amendment will create no serious burden for the committtees”. (link)

OMBWatch also mentions in their note that multiple formats for proceedings are vastly preferred to the one-of-the-above approach that S1 requires.  Not only does this make it easier to watch, digest, quote, or share, but this also will make the committees more likely compliant with the section 508 accessibility standards, giving citizens, staff, and members with disabilities access to records of proceedings.  (Patrice McDermott of OpenTheGovernment.org has also vocally supported robust committee disclosure requirements.)

Finally, our discussion of implementing S1 has led us to realize that new standards for posting public information online lead inevitably to new challenges in digital records management and preservation.  If the committee Web sites become the go to source for committee related information (where before there was no digital source), then who becomes responsible for this digital history?  Committee documents become the property of the National Archives (specifically the Center for Legislative Archives) after each Congress.  As I observed in the previous discussion of this topic, it may end up being easier to get committee documents online than it will be to get them to stay there.  Ideally, I think committees should probably maintain jurisdiction over their documents, and have an easy procedure to link to an archive of previous committee procedures.

Kudos to the Finance Committee (and especially Senator Salazar) for getting the proceedings requirement introduced, and for following up more than year later.

For more background on the document, see this post, and for updates, I’m keeping notes on this page, from which future updates will likely be pulled.

-John  (start review)

foreword:
compiled by the Task Force on the Documentation of Congress of the Society of American Archivists Congressional Archivists Roundtable, coming from 1989’s “Understanding Congress: A Bicentennial Research Conference”.  “the fragmented nature of congressional primary source documentation” is partly responsible for the lack of scholarly writing on the legislative branch.  Report is a “study of the archival sources that document the operations of Congress.”

Preface:
“Because the documentation of Congress, in particular, most directly reveals the will of the people as expressed through their elected representatives, it is especially crucial to preserve evidence and information about the legislative process and make it accessible to the public.”  (gives great detail on problems with public access: “fragmented and geographically scattered; collections are often voluminous, of complex arrangement, inadequately indexed, and in poor physical condition; contents of many collections are uneven, with unexplained gaps in information; and repositories that receive these collections frequently lack the resources to provide state-of-the-art arrangement, description, and archival preservation.”  Project undertaken by the Task Force on Congressional Documentation of the Society of American Archivists’ Congressional Archivists Roundtable (that’s correctly transcribed).  “many of its suggestions will take years to be carried out; others can be effected immediately.”  “Among the most pressing needs are actions to improve the documentation of legislation, representation, congressional leadership, political activities, and programs of congressional support agencies.  Other recommendations are aimed at better documenting Congress’ relations and interaction with media, the executive and the judicial branches, lobbyists, and think tanks.  Finally, steps are suggested to improve documentation of the administration of Congress’ to fill gaps in the historical record through structured, coordinated oral history interview programs; and to improve the preservation of congressional sources.

Intro:
Report organized into congressional “functions”, “documentation”, and “recommendations”.  (apparently there’s a 1978 report from the National Study Commission on the Records and Documents of Federal Officials, which “recommended that office files and personal papers of members of Congress be legally designated as federal records with guaranteed public access after fifteen years” (this wasn’t implemented, but that report would be useful to find).  This report led to a 2 day conference, where they decided to publish a handbook on member records management.  1985 then saw a 2 day conference on documenting Congress, put on by the “Dirksen Congressional Center and the national Historical Publications and Records Commission” (another great report to find)  This conference led to the creation of the Congressional Archivists Roundtable of the Society of American Archivists (which, in turn, led to the current day 2008 Advisory Committee on the Records of Congress (link).  (The Congressional Archivists Roundtable appears to be defunct, perhaps being defunded in the mid 90’s?)  “the “importance” [of congressional material] had not led to a determined effort to systematically appraise and preserve a documentary record of Congress.” (page vi)  Problems archivists face: “the information explosion, the computer and telecommunications revolutions, insufficient resources for archival work as government and repository budgets tighten, and the lack of clearly defined long-term strategies and action plans to accomplish overall documentary objectives.”  The response to this is a strategy, which matches the organization of this document (functions, documentation, and recommendations).    The report then lists participants.

Page ix lists contents, listing the major topics to be examined: Institutional Setting, the Legislative Process, Representation, Political Activities, External Relations, Administration and Support, Research Use of Congressional Collections, Appendices, and Notes.

Summary Report and Recommendations
Repeats problem.  “Historical records do not simply materialize.”  They’re trying to balance the needs of three authorities: members and officials of Congress “individually responsible for the on-site management of the information that is collected and maintained in their offices”, NARA’s CLA, and the “literally hundreds of archival repositories across the country [that] preserve and provide access to the personal papers that are deposited in them by the members”.  This listing seems to me to be a result of their institutional setting, my take on the jurisdictions at work in congressional information access can be found here.

Major Findings:
1. “Congressional committees are relatively, although not uniformly, well documented, [but] there is great variation in the documentary quality of individual members’ collections”  This conclusion strikes me as a result of writing in the mid 1990s, when paying meaningful attention to legislative affairs through the Internet was rather impossible.  There was little difference then between “well documented” and “publicly available (online)”, where now that difference is quite clear.  What is “well documented”, like committee hearing transcripts, or the upcoming schedules for committee hearings, in the archival sense, can also be useless for those hoping to actually watch Congress in action, even in time for upcoming elections, which is a much lower bar than the near-real time awareness lobbyists need in order to be legislatively relevant.  Thus the report’s focus on member records, which were managed in a much less standard manner then.

2. The need for a “coordinated retention plan that meets the long-term needs of Congress” as applied to congressional support agencies.  The GAO was doing well, the (now defunct) Office of Technology Assessment and Government Printing Office had partial programs, and the CRS and CBO had none.  Again, no real mention of OTA and GAO providing public documentation while CRS does not.  Our expectations of finding things online has led to a new set of expectations.  (well, that and the expectation of equal access to publicly funded documents, since CRS reports are sold through private companies.)

3. Executive Branch and Judicial Branches are doing a rather good job, but are relevant here nevertheless.

4. Other sources they feel have been thitherto overlooked: nat’l, congressional, and individual campaign committees, political party organizations; and congressional member organizations and caucuses.  (still true, 16 years later)

5. Member documentary repositories are hard to use, recommend better practices here.

6. Member materials will be better processed if offices hire and train archivists, and keep up with documents processing.

The Report then launches into specific recommendations, lining up with the table of contents, but in summary form.

The preface describes it as a “study of the archival sources that document the operations of Congress.”  The “archival sources” described in this document comprise the entire body of public congressional information, the substance of both administrative minutiae, and legislative substance.  Just as we are interested in the capacity of the public to be conscious of its legislature, we should be interested in the legislature’s capacity to take stock of itself, to engage in constructive introspection.

I came across this document being repeatedly cited while reading the yearly reports of the Advisory Committee on the Preservation of the Records of Congress, and still find rich irony in the fact that the document itself wasn’t available in a digital form.  That’s not to say anything against the Advisory Committee, which seems to be an outgrowth or a result of the task force that wrote S.Pub 102-20, and also inspired H.R. 5241 from the 101st Congress, a bill reorganizing the National Archives, among other things.  The Advisory Committee seems to be among the very best of examples of an organization created to meet an emergent need, cutting across jurisdictions and what one of its members recently described to me as “negotiated terrain” (a description I very much liked).

The complex problem of coordinating congressional information is difficult, but not for the usual reasons.  As far as preservation goes, the administrative coordination is already in place, and it seems that the research (and even enforcement ) about disclosure mechanisms has been in place for quite some time.  What has been lagging is not administrative will, but the digital culture and popular expectations that make IT investment a real priority.

This is clearly changing, as new staffers expect to represent their members of Congress online without encountering arcane restrictions, as citizens expect to encounter government information and services through the same search engines they use for research and shopping, and a new brand of journalism is springing up that depends not on cultivating trusted sources through personal relationships, but on careful consideration of primary sources–exactly those “archival sources” this document so comprehensively describes.

While some disclosure will be resisted for as long as the benefits of secrecy outweigh the outcry over obstruction, and privileged access will always be at odds with the broader public interest, it is good to see that a detailed anatomy of congressional information has already been constructed in great detail.  The question that remains is how well will Congress adapt to new expectations of information access — a question that necessarily comes along with a digitally empowered citizenry.

Important links:

A new website www.opengovdata.org came out of it, which has nice announcement text

as well as a wiki wiki.opengovdata.org (which I’m hosting, so blame me for problems) for ongoing discussion on neutral turf.

There’s a Flickr tag with a bunch of photos. You can see that Tim O’Reilly’s big colored sticky note cards played an important role in many sessions.

One of the tangible results of the conference was a set of eight principles for how to determine whether some government data is “open”. It’s similar to how we use criteria elsewhere to determine whether software is open, and also the Open Knowledge Definition. And it was suggested that we develop some sort of branding that we all can make use of to support and point to the principles. The discussion pages linked from some of the terms in the principles are editable wiki pages and do need to be fleshed out with suggestions from anyone.

Also, Dan Newman started some discussion about how to mobilize citizens at large over transparency issues. I am eager to see how that discussion continues— I expect some organizing will happen on the (open) mail list created at the conference (and linked from www.opengovdata.org; yes, yet another mail list…).

The collection seems well organized and easy to peruse, with solid explanations of their methodology and disclaimers about what’s available based on the crawling.

My main suggestion is that the archiving happen with greater frequency, perhaps coordinated in order to capture the greatest amount of material possible, and for those responsible for the Web Harvest to coordinate with the CAO, systems administrators, and vendors to be sure that the digital records management practices used in organizing member sites encourages easy crawling and archiving by NARA and CLA.

The House has a document laying out best practices for documents management for House offices; I wonder if the digital materials management should be expanded to include digital materials availability, perhaps including standards like sitemapping, in order to ensure the preservation of member sites?

My other suggestion is to increase the exposure of the captured sites, perhaps encouraging links from the bioguides, or current member sites, and to ensure that the collection itself is crawlable through search engine indexing practices.