Researching history has always been a passion of mine. It’s something that I’ve always gone back to no matter what’s going on in my life, and through this hobby I’ve always enjoyed digging into some obscure bit of history and learning everything I can about it. Working with primary documentation and actually putting my hands on original source material is one of the most gratifying ways to indulge this passion of mine. As someone who now works professionally with advanced technologies, I’m constantly looking for new ways to bring technology into parity with the activities I participate in, and recently, I’ve found a way to improve the research I do in my off time.
When you get into historical research, especially when that overlaps with military and governmental activities, you quickly end up needing to learn how to find the documents that you need. Oftentimes that means knowing where the right archives are, and sometimes that involves knowing where to go to get the right documents, knowing who to ask, and most importantly, knowing how to ask. That’s where FOIA comes in.
In the 1960s, the US federal government (including many state and local governments over the years) rolled out a plan to improve government transparency called the Freedom of Information Act. The idea behind FOIA was to allow the citizens of the country to ask questions of their government and get direct, documentary-backed answers to their queries. Some agencies are better than others at being responsive to these requests for information, but the bottom line is that all federal agencies are required by law to be responsive to these requests to the extent that national security won’t be harmed by the answers of these requests. What this act has given us is an incredible amount of insight into how the government functions including large deposits of unclassified documents from 3-letter agencies with a history that they sometimes don’t like to own up to.
But with any system like this, there are quirks, flaws, and issues related to the types of documents that are released, and more specifically how they are released. Take the Central Intelligence Agency for example. They tend to be fairly good about publishing material that pertaining to their history and make it available to the public through a “reading room” that anyone can go to and search through their databases for information they may be researching. They call it the “CREST” database. Now the thing about CREST is that when you search for something, it’s not always clear how you got the type of document you were looking for. Sometimes you can get erroneous or irrelevant documentation that somehow relates to your search term. It’s enough to wonder what kind of technology they have to work with and if they specifically chose weaker tech in order to deliver spottier results.
Whatever the reason, they do provide the public with clear documentation straight from their archives which helps researchers and historians do their work. The biggest problem with this is that the agency delivers these documents in flat PDF files – I.E. unsearchable documents which are sometimes hundreds of pages long.
In the next entry on this series, I’ll go into how we can bring technology to this issue and help solve a tough problem that faces researchers today.