We have recently been called upon by Congress in the joint explanatory statement on the H.R. 1105, to work with the Library of Congress, including the Congressional Research Service, and the Law Library of Congress, to discuss access to bulk data. Specifically, the language is as follows:

[JT: omitted --- I've posted it before here]

To address this request, a Legislative branch task force has been assembled consisting of representatives from the offices of the Secretary of the Senate, the Clerk of the House, the Library of Congress, Congressional Research Service, the Law Library of Congress, and GPO. This task force has already met and is working to develop a position on access to bulk data. We will look to this work and the review by Congress to help guide our work on making bulk data accessible.

Grin.

The two paragraphs were added by Congressman Mike Honda of California, one of our champions of the use of technology to further transparency and civic engagement. John Wonderlich of Sunlight Foundation, Rob Pierson in Honda’s office, and I collaborated on this over a long period of time. Honda got involved in 2007 asking the Library to look into this and then in 2008 getting the paragraphs added to the bill markup.

So here they are:

Congressional Technology Coordination.-The House of Representatives needs a strategic and coordinated plan that will prepare for the future technology needs of the institution. A 2006 report commissioned by the Chief Administrative Officer and the Committee on House Administration, entitled Strategic Technology Road Map for the Ten Year Vision of Technology in the House of Representatives, provided a suggested structure for Information Technology evaluation and decision making. The Chief Administrative Officer, the Clerk, and the Sergeant at Arms are asked to prepare a report by June 30, 2009 on their efforts or plans to develop House-wide data-sharing standards; implement standard legislative document formats; address the increasing resource challenges of Member offices; and identify disparate systems throughout the institution that prevent it from taking advantage of economies of scale. [page 2]

and

Public Access to Legislative Data. There is support for enhancing public access to legislative documents, bill status, summary information, and other legislative data through more direct methods such as bulk data downloads and other means of no-charge digital access to legislative databases. The Library of Congress, Congressional Research Service, and Government Printing Office and the appropriate entities of the House of Representatives are directed to prepare a report on the feasibility of providing advanced search capabilities. This report is to be provided to the Committees on Appropriations of the House and Senate within 120 days of the release of Legislative Information System 2.0. [page 11]

According to an article in Wired: “In our web 2.0 world, we can empower the public by providing them with raw data that they can remix and reuse in new and innovative ways,” says Honda, who is vice chairman of the Appropriations Subcommittee on the Legislative Branch. “With these tools, the public can collaborate on projects that can help legislators to create better policies to address the pressing challenges facing our nation.” There’s also a good article at Mother Jones and a nice post by Tim O’Reilly.

The concept of bulk data downloads hasn’t been missed by many parts of the government. The Census Bureau and the Federal Elections Commission, for instance, are fantastic at sharing with the public as much as they can. In the latter case it is electronic versions of campaign contribution filings, which is obviously very important for preventing corruption. But, there are significant gaps in other areas of the government where a little legislating is necessary. Here we’re talking about information on bills in congress going back around two decades, and the information going forward.

The Library of Congress has a database of this information but they don’t share it with the public. Sharing it would mean that creating sites like GovTrack — and the various other sites that use data from GovTrack including OpenCongress and MAPLight.org — would be a little easier, but also a little more accurate. Right now GovTrack goes through a roundabout process to reverse-engineer the same information we are seeking from this database. Basically, we already have the information by scraping it off of thomas.loc.gov — we’d just rather get it directly rather than the way it is assembled now. So because I go through so much trouble to reverse-engineer the data I want, not so many things will change in an obvious way on GovTrack — it’ll just be that my life will be a little easier and the information will be a little more complete and up to date. But, you can expect to see other sites spring up doing new and interesting things with the information — ways of visualizing the congressional process that we couldn’t yet imagine.

The Government Printing Office is mentioned because of how they make legislative documents like the text of bills available to the public. PDFs and text-only versions are made available for free already. No problem there. But they have other files that would be useful to sites like GovTrack which they sell at ridiculously high subscription prices. Those files would make comparisons of bill text easier to produce (although GovTrack already has this feature, again by essentially going about it the hard way). If you think about it from the perspective that some bills go through Congress so fast no one has time to read them through, being able to apply technology to the process is so important, like to detect changes in the text of bills between versions to make it easier for people to get through it. This is what GPO is preventing by selling some of its files, rather than providing them to the public for free (which it is essentially mandated to do for most documents — why they exempt certain documents is not known).

Now, it’s not that the Library doesn’t necessarily *want* to share its database. It’s just that sharing it wasn’t a part of their mandate from Congress and they don’t want to upset Congress by stepping out of their mandate. The omnibus bill is an indication from the House to the Library that this would be something supported by Congress. (My understanding is that the Library has been seeking permission from Congress to do some of these things, probably in response to a previous push for this, but the omnibus legislation has been in the works concurrently.)

His office also submitted a second paragraph which I’ll get to below.

Rob Pierson in Honda’s office writes on the OHP mail list:

I’ve mentioned on the list some of the steps my boss (Congressman Honda) has been taking, with counsel from many folks on this list, to guide Congressional policies on the path towards effectively leveraging technology to open up access to the public. There are actually quite a few other staffers who also follow this list, and we’ve certainly learned quite a bit from the conversations posted here, so I wanted to throw out a quick note of appreciation to everyone who has been contributing to the discussions.

With guidance from the conversations on this list (and the OHP report), Congressman Honda recently submitted the following sections into the House Legislative Branch Appropriations Report. The following (or possibly very similar versions) were included in the Leg Branch Subcommittee markup of the bill:

*Public Access to Legislative Data (as submitted)*

The Committee believes that the public should have improved access to legislative information through more advanced search capabilities such as those available through the Library of Congress’ Legislative
Information System. The Committee also supports enhancing public access to legislative documents, bill status, summary information, and other legislative data, through more direct methods such as bulk data downloads and other means of no-charge digital access to legislative databases. The Committee requests that the Library and Government Printing Office report on the progress towards these goals within 90 days of enactment of this Act.

Note that the GPO has also been stuck in there. More more on that, see this post.

The second paragraph that Honda’s office submitted John noted was parallel to the final chapter of our report, Coordinating Web Standards. (Hmm, I principally wrote that chapter too….)

*Congressional Technology Coordination (as submitted)*

The Committee recognizes the need for the House of Representatives to develop a strategic and coordinated plan that will prepare for the future technology needs of the institution. A 2006 report commissioned by the Chief Administrative Officer and the Committee on House Administration, entitled /Strategic Technology Road Map for the Ten Year Vision of Technology in the House of Representatives/ provided a suggested structure for an IT evaluation and decision-making process.
No later than 90 days after the enactment of this Act, the Committee requests that the Chief Administrative Officer, the Clerk, and the Sergeant at Arms report to the Committee of their efforts to develop House-wide data-sharing standards; implement standard legislative document formats; address the increasing resource challenges of Member offices; and identify disparate systems throughout the institution, which prevent it from taking advantage of economies of scale.

This is of course fantastic news for anyone that supports transparency, which is, well, everyone in their right mind, I think. So thanks to Congressman Honda for taking the initiative on this!

(Other links: last year’s leg branch appropriations blog post, my first or one of my first posts here about structured data)

For more background on the document, see this post, and for updates, I’m keeping notes on this page, from which future updates will likely be pulled.

-John  (start review)

foreword:
compiled by the Task Force on the Documentation of Congress of the Society of American Archivists Congressional Archivists Roundtable, coming from 1989’s “Understanding Congress: A Bicentennial Research Conference”.  “the fragmented nature of congressional primary source documentation” is partly responsible for the lack of scholarly writing on the legislative branch.  Report is a “study of the archival sources that document the operations of Congress.”

Preface:
“Because the documentation of Congress, in particular, most directly reveals the will of the people as expressed through their elected representatives, it is especially crucial to preserve evidence and information about the legislative process and make it accessible to the public.”  (gives great detail on problems with public access: “fragmented and geographically scattered; collections are often voluminous, of complex arrangement, inadequately indexed, and in poor physical condition; contents of many collections are uneven, with unexplained gaps in information; and repositories that receive these collections frequently lack the resources to provide state-of-the-art arrangement, description, and archival preservation.”  Project undertaken by the Task Force on Congressional Documentation of the Society of American Archivists’ Congressional Archivists Roundtable (that’s correctly transcribed).  “many of its suggestions will take years to be carried out; others can be effected immediately.”  “Among the most pressing needs are actions to improve the documentation of legislation, representation, congressional leadership, political activities, and programs of congressional support agencies.  Other recommendations are aimed at better documenting Congress’ relations and interaction with media, the executive and the judicial branches, lobbyists, and think tanks.  Finally, steps are suggested to improve documentation of the administration of Congress’ to fill gaps in the historical record through structured, coordinated oral history interview programs; and to improve the preservation of congressional sources.

Intro:
Report organized into congressional “functions”, “documentation”, and “recommendations”.  (apparently there’s a 1978 report from the National Study Commission on the Records and Documents of Federal Officials, which “recommended that office files and personal papers of members of Congress be legally designated as federal records with guaranteed public access after fifteen years” (this wasn’t implemented, but that report would be useful to find).  This report led to a 2 day conference, where they decided to publish a handbook on member records management.  1985 then saw a 2 day conference on documenting Congress, put on by the “Dirksen Congressional Center and the national Historical Publications and Records Commission” (another great report to find)  This conference led to the creation of the Congressional Archivists Roundtable of the Society of American Archivists (which, in turn, led to the current day 2008 Advisory Committee on the Records of Congress (link).  (The Congressional Archivists Roundtable appears to be defunct, perhaps being defunded in the mid 90’s?)  “the “importance” [of congressional material] had not led to a determined effort to systematically appraise and preserve a documentary record of Congress.” (page vi)  Problems archivists face: “the information explosion, the computer and telecommunications revolutions, insufficient resources for archival work as government and repository budgets tighten, and the lack of clearly defined long-term strategies and action plans to accomplish overall documentary objectives.”  The response to this is a strategy, which matches the organization of this document (functions, documentation, and recommendations).    The report then lists participants.

Page ix lists contents, listing the major topics to be examined: Institutional Setting, the Legislative Process, Representation, Political Activities, External Relations, Administration and Support, Research Use of Congressional Collections, Appendices, and Notes.

Summary Report and Recommendations
Repeats problem.  “Historical records do not simply materialize.”  They’re trying to balance the needs of three authorities: members and officials of Congress “individually responsible for the on-site management of the information that is collected and maintained in their offices”, NARA’s CLA, and the “literally hundreds of archival repositories across the country [that] preserve and provide access to the personal papers that are deposited in them by the members”.  This listing seems to me to be a result of their institutional setting, my take on the jurisdictions at work in congressional information access can be found here.

Major Findings:
1. “Congressional committees are relatively, although not uniformly, well documented, [but] there is great variation in the documentary quality of individual members’ collections”  This conclusion strikes me as a result of writing in the mid 1990s, when paying meaningful attention to legislative affairs through the Internet was rather impossible.  There was little difference then between “well documented” and “publicly available (online)”, where now that difference is quite clear.  What is “well documented”, like committee hearing transcripts, or the upcoming schedules for committee hearings, in the archival sense, can also be useless for those hoping to actually watch Congress in action, even in time for upcoming elections, which is a much lower bar than the near-real time awareness lobbyists need in order to be legislatively relevant.  Thus the report’s focus on member records, which were managed in a much less standard manner then.

2. The need for a “coordinated retention plan that meets the long-term needs of Congress” as applied to congressional support agencies.  The GAO was doing well, the (now defunct) Office of Technology Assessment and Government Printing Office had partial programs, and the CRS and CBO had none.  Again, no real mention of OTA and GAO providing public documentation while CRS does not.  Our expectations of finding things online has led to a new set of expectations.  (well, that and the expectation of equal access to publicly funded documents, since CRS reports are sold through private companies.)

3. Executive Branch and Judicial Branches are doing a rather good job, but are relevant here nevertheless.

4. Other sources they feel have been thitherto overlooked: nat’l, congressional, and individual campaign committees, political party organizations; and congressional member organizations and caucuses.  (still true, 16 years later)

5. Member documentary repositories are hard to use, recommend better practices here.

6. Member materials will be better processed if offices hire and train archivists, and keep up with documents processing.

The Report then launches into specific recommendations, lining up with the table of contents, but in summary form.

More background discussion can be found here.

Congressional Budget Office (CBO): CBO intro PDF from their website, see also Congressional Budget Act of 1974

General Accountability Office (GAO):

Library of Congress (LOC):

Government Printing Office (GPO)

National Archives and Records Administration (NARA)

Chief Administrative Officer (of the House) (CAO)

Speaker of the House (link)

Secretary of the Senate (link)

Senate Rules Committee (link)

Senate Sergeant at Arms (link)

Committee on House Administration (CHA)

Joint Committee on Printing (JCP)

Senate Homeland Security and Government Affairs Committee (HSGA)

House Committee on Oversight and Government Reform (link)

Advisory Committee on the Records of Congress (link)

Senate Historical Office (link)

Legislative Resource Center (LRC )

Executive Branch

General Services Administration (GSA), (executive branch)

Office of Management and Budget (OMB)

Judicial Branch

Federal Judicial Center (FJC)

Transparency bills that passed in 110th Congress, 1st session:

1. S.1 Honest Leadership and Open Government Act of 2007:

2. S.2488 OPEN Government Act of 2007:

3. OTA funding implemented as part of GAO:

Other developments during 110th Congress, 1st session:

1. C-SPAN changed its copyright policy to allow reproduction on the Web
2. Seven Members of Congress posted daily schedules to their official Web sites
3. Two Members of Congress posted their personal financial disclosures to their official Web sites
4. The number of Members blogging and posting their earmarks increased dramatically
5. Launch of OMB spending and earmark online database
6. Launch of FARA online database
7. Redesign of SOPR lobbyist disclosure online database

Pending for 110th Congress, 2nd session:  (see also here )

1. S.2321 E-Government Reauthorization Act of 2007 (Hearings held in the Senate)
2. Review of Franking Regulations for Member Web sites
3. H.R.1362 Accountability in Contracting Act of 2007 (Passed the House)
4. S. Res. 401 A resolution to provide Internet access to certain Congressional Research Service publications
5. Lieberman’s statement that Senate votes should be in XML, in support of THOMAS upgrades

Two more pieces of transparency reform legislation underway for 2008:
First, the Presidential Library Donation Reform Act of 2007, (HR
1254<http://www.govtrack.us/congress/bill.xpd?bill=h110-1254>)
saying <http://www.govtrack.us/congress/billtext.xpd?bill=h110-1254>… *”Any
Presidential library fundraising organization shall submit on a quarterly
basis, in accordance with paragraph (2), information with respect to…”
*Second, here’s the Presidential Records Acts Amendments of 2007, (HR
1255<http://www.govtrack.us/congress/bill.xpd?bill=h110-1255>)
saying <http://www.govtrack.us/congress/billtext.xpd?bill=h110-1255>…
“`(a)(1) When the Archivist determines under this chapter to make available
to the public any Presidential record that has not previously been made
available to the public, the Archivist shall…” and “Executive Order No.
13233, dated November 1, 2001 (66 Fed. Reg. 56025), shall have no force or
effect.” among other things.  (link to executive
order<http://www.whitehouse.gov/news/releases/2001/11/20011101-12.html>
)
Again, the OHP page<http://www.theopenhouseproject.com/resources/pendingbills>also
has legislation on Congress relevant to the report as well.

It’s an awesome collection of about 3,000 images, of the quality you’d expect from the world’s largest library.  It’s wonderful to see them available the same way we expect to share images with each other, sort of making history less of something living in a museum, and more of something available, relevant, and even sorted through tags.

The reason this is interesting to me is that we like to capture reality in data. The Library of Congress and GovTrack both systematize (or in computer jargon “normalize”) the bill-becomes-a-law process. At every point in the game, a bill, in our data formats, is either in-progress, enacted, dead, etc. It must be in one of these states. After all, the constitution outlines exactly what states a bill can be in, so any bill *must* be in one of these states.

But if we’re not sure what state a bill is in, what state do we put it in in our data? There’s also the more important question- What do the lawmakers do if they disagree about what state a bill is in? (Actually, I would prefer to phrase it as “what state they are in”, but that’s another story.) Wikipedia describes (what the editors of the page claim is) a current debacle over H.R. 1585: National Defense Authorization Act FY 2008:

In December of 2007, President George W. Bush pushed the pocket veto into murky waters by claiming that he had pocket vetoed H.R. 1585, the “National Defense Authorization Act for Fiscal Year 2008,” even though the House of Representatives had designated agents to receive presidential messages before adjourning. The bill had been previously passed by veto-proof majorities in both the House and the Senate [JT: and thus a traditional veto would have been futile].

So was the bill (pocket) vetoed or not? Is the bill still in-progress? Assuming it was not pocket vetoed, after 10 legislative days without a traditional veto it becomes law, and us citizens would hate to be on that 11th day without either resolution on the pocket veto matter or a traditional veto, because then we as a country will not know whether this bill has become law. (Another question: How might the Supreme Court assert jurisdiction over this question.)

But back to the data. At one point, some time after Dec. 28, someone in the House responsible for updating the bill status information shown on THOMAS entered a new status line:

Dec 28, 2007: Pocket Vetoed by President.

GovTrack picked up on the change and shows that status currently, much to the confusion of several people emailing me about it. Looking back at THOMAS, it seems like someone realized that that was apparently quite a constitutional (if not political) claim and retracted that update, because it not longer says that.

In many cases citizens complain when the government takes things back, hiding information previously made public. That’s definitely not what I am getting at here. THOMAS is forced to show *something*, and when it doubt… well, what can you do but roll back history until we figure out what the next legislative step actually *was*.

First, a report summarizing lobbying reform measures:

Significant changes were made by Congress to the current lobbying laws, and to internal House and Senate rules on ethics and procedures, by the passage of S. 1, 110th Congress, and the adoption of H.Res. 6, 110th Congress. In the face of mounting public and congressional concern over allegations and convictions of certain lobbyists and public officials in a burgeoning “lobbying and gift” scandal, and with a recognition of legitimate concerns over undue influence and access of certain special interests to public officials, Congress has adopted stricter rules, regulations, and laws attempting to address these issues. This report examines the changes made to law and congressional rule in S. 1, 110th Congress, and changes adopted to internal House rules earlier in the Congress in H.Res. 6.

The other report is a brand new analysis of Iraq: Iraq: Post-Saddam Governance and Security.:

Operation Iraqi Freedom overthrew Saddam Hussein’s regime, but much of Iraq remains violent because of Sunni Arab resentment and a related insurgency, compounded by Sunni-Shiite sectarian violence that, in the judgment of many, constitutes a “civil war.” Mounting U.S. casualties and financial costs — without dramatic improvements in levels of violence or clear movement toward national political reconciliation among Iraq’s major communities — have intensified a debate within the United States over whether to reduce U.S. involvement without completely accomplishing initial U.S. goals. President Bush announced a new strategy on January 10, 2007 (“New Way Forward”) consisting of deployment of an additional 28,500 U.S. forces (“troop surge”) to help stabilize Baghdad and restive Anbar Province. The strategy is intended to provide security conditions conducive to Iraqi government action on a series of key reconciliation initiatives that are viewed as “benchmarks” of political progress.

I find the Statements of Administrative Policy to be particularly helpful, since the administration’s official policy positions on various legislative initiatives are all available as they’re available.

Now if bills as they’re reported out of committee and considered on the floor were to get assigned a URI and everyone used a standardized version of LOC’s Legislative Indexing Vocabulary, then these disparate data sets could all be more easily linked for centralized tracking.