While this strikes me as a great development, since increasing access to public information should only increase its usefullness and impact, this also raises questions to me.

It strikes me that this kind of cloud computing (which I learned about at Princeton’s CITP Cloud Computing event) will start to affect the way we think about what is a public utility. New kinds of relationships will exist between established institutions and new “cloud” service providers, which come with new opportunities for gain, abuse, conflict of interest, unseen liabilities, etc.

For example, I expect that Google will be able to see all sorts of interesting metadata about who links to specific Hubble images, or who queries scientific databases, or how. The question, then, is whether that sort of information will be publicly available (or even if it could be). If not, then Google’s benevolence starts to look a lot more like self interest, where they gain not only by becoming the arbiter of the public’s access to their information stores, but also by gaining a privileged view of how we relate to our public data.

This isn’t an isolated academic question, either. The way research data are cited and linked is itself the subject of scientific inquiry, will certainly continue to be invaluable.

Perhaps this is gift-horse-mouth looking, and we should be glad that someone wants to provide a free accessible home to public data. A little cynicism however, seems in order, and we might have to rethink what it means to provide a free public service.

The collection seems well organized and easy to peruse, with solid explanations of their methodology and disclaimers about what’s available based on the crawling.

My main suggestion is that the archiving happen with greater frequency, perhaps coordinated in order to capture the greatest amount of material possible, and for those responsible for the Web Harvest to coordinate with the CAO, systems administrators, and vendors to be sure that the digital records management practices used in organizing member sites encourages easy crawling and archiving by NARA and CLA.

The House has a document laying out best practices for documents management for House offices; I wonder if the digital materials management should be expanded to include digital materials availability, perhaps including standards like sitemapping, in order to ensure the preservation of member sites?

My other suggestion is to increase the exposure of the captured sites, perhaps encouraging links from the bioguides, or current member sites, and to ensure that the collection itself is crawlable through search engine indexing practices.

There is a clear public benefit, however,when search terms like “Colorado census 1990″, “federal childhood immunology standards”, “Pennsylvania superfund sites”, or “Congressional Record 1930 Stock Market” result in the information the searcher is obviously interested in–government information.

The solution to this problem is a non-proprietary standard championed by Google, called the sitemap protocol. Implementing this standard helps automated web-crawlers (the stuff of search engines) find their way around your entire site. Given the immense number of government databases and agencies to reach, getting all government information to show up in web searches will take some time–unless you have a flexible bureaucracy, or an administrative commitment to modern digital government.

Enter Google’s public policy team, and the Senate Homeland Security and Government Affairs Committee:

The Senate Homeland Security and Government Affairs Committee will consider S. 2321, which extends and updates the E-Government Act of 2002. Part of the bill directs the Office of Management and Budget to create guidance and best practices for federal agencies to make their websites more accessible to search engine crawlers, and thus to citizens who rely on search engines to access information provided by their government. It also requires federal agencies to ensure their compliance with that guidance and directs OMB to report annually to Congress on agencies’ progress.

From the bill:

`(i) GUIDELINES- Not later than 1 year after the date of enactment of the E-Government Reauthorization Act of 2007, the Director shall promulgate guidance and best practices to ensure that publicly available online Federal Government information and services are made more accessible to external search capabilities, including commercial and governmental search capabilities. The guidance and best practices shall include guidelines for each agency to test the accessibility of the websites of that agency to external search capabilities.

This measure will go a long way toward making governmental information relevant to people’s lives, by making them accessible in the places we would first expect to find them.