1. Inventory and Analysis of a Digital Accession¶

One of the early steps in taking in a digital accession is to perform some kind of appraisal, including an inventory or survey of the materials. The survey will inform your next steps for caring for the deposit, including formulating an appropriate preservation plan. Hopefully this is aided by some kind of inventory and an interview provided by the donor, but often it is the case that a donor does not leave any inventory. When accessions are relatively small, we can perform an inventory by merely browsing the contents and making some notes on the format and directory structure of the files. However, when the number of files jumps into the hundreds, thousands, or tens of thousands, we need to reach for more powerful tools. This notebook demonstrates in Python code some methods for building an inventory for a set of files and then some code to summarize and visualize accession contents. Many of the same principles at work in these examples are at work in larger preservation tools or environments, such as BitCurator, or Preservica, but you may find them useful as complementary tools.

Issues an Inventory May Uncover¶

The inventory and summary may reveal the presence of file formats that have not been encountered previously by your institution. Such file formats may be unique to the donor's discipline or trade, or they may be unique to a particular researcher's practice. It is important to understand how these files are produced and the intellectual works within them, but first you have to find them. Folder structures are also sometimes standardized for a particular tool or practice. These folder structures are laid out according to a pattern that serves the workflow in which they were produced or managed. Folder structures are often used to create websites, geographic databases, audiovisual productions, or software projects.

You will also want to look for these issues, but they are beyond the scope of this notebook:

• Folders or files that were not intended for deposit.
• Private or embarassing material.
• Computer viruses

Step One: Buildng the Inventory¶

The first step is to build a basic file inventory for a sample accession, using Python code. The Python code below collects information about every file within the samples folder, including subfolders and so on.. The details about each file are appended to a comma-separated values (CSV) text file. You can point the inventory code at any accession folder to build an inventory CSV file.

The inventory code records this information:

• file name
• file location - This is the folder path where the file lives.
• file extension - This is the part of the filename after the period, such as "PDF" or "docx".
• file format - This is the two-part MIME Type for the file. (More detail to follow)
• file size - This is the size of the file in bytes.
• modification timestamp - The date and time of the last time this file was modified.

Inspect the Raw Inventory CSV¶

After you execute the file inventory code above, you will see a new CSV file in your notebook directory. If you like, you can open that inventory CSV file in Jupyter, which will display it in tabular form. The next block of code will display the first few lines of the file as raw text.

You can see above that the first line of the CSV file is made up of column names. The data then follows, row by row.

Next we will load the same CSV data into a Python data science tool called Pandas. This means that Python will read your file into a Pandas "data frame" object that we will use going forward. Data frame objects hold tabular data for further processing and display. We will first print the same first few rows of data to give you a sense of how it is organized and let you see that it is working properly. We will also call the shape() function on the data frame object, which gives us the exact dimensions (row and columns) in the data frame.

Step Two: Visualize the Inventory¶

The next step is to use some powerful Python tools to show us the inventory in a variety of way. These tools are not installed by default in Jupyter environments, so you will first have to run the following code block to install them.

Visualize Files by their Media Type¶

Our next step is to visualize the file format information in aggregate, so that we can make preservation decisions. Here we look at a summary of the media types using a standard called the IANA MIME type. IANA MIME types have a type and a subtype, separated by a slash (/) character.

type/subtype

For example, these are some common MIME types:

image/jpeg
text/plain
application/pdf

Sometimes MIME types are longer and more complex. Here you can see the difference between older MS Word documents and the more recent .docx documents:

application/msword
application/vnd.openxmlformats-officedocument.wordprocessingml.document

In the visualization below the area of each block reflects the number of files of each type and subtype. The color of each block represents the number of bytes stored in each type and subtype. This kind of visualization is called a Treemap and they were first developed in the early 90's by Ben Shneiderman in the Human-Computer Interaction Lab at the University of Maryland's iSchool.

These treemaps are interactive; with some detailed values presented when you hover your pointer on a block and a zoom-in feature when you click on a block.

Exercise: Mimetype Analysis¶

Look over the treemap above and notice which types of files are the most common (larger boxes). What conclusions can you draw about this accession based on the common file types? What does the treemap tell us is significant about the PDF files in this accession?

File Extension vs. Mimetype¶

A quick look at the same visualization for file extension (the part of the filename after the last dot) shows why we use mimetypes. File extensions are not always informative, whereas at least mimetypes give us a high-level category for each format.

Python's mimetype guessing module is useful for a quick analysis, but there are more sophisticated file format identification tools that you will find helpful. Tools such as Droid and Siegfried (recommended) offer the ability to identify detailed sub-formats for the files in your accessions, such as which version of PDF was used to produce a document. These tools require a fully functional computer with adminitrator access to install and so they are beyond the scope of this notebook demonstration, but they will catch a many more specific formats than Python.

Exercise: Comparing File Extensions with Mimetype¶

The treemap above shows the distribution of files, by area, and bytes, by color, across file extensions. How does it differ from the treemap showing the same distributions across mimetypes? What information is missing?

Identify Important Folders¶

This treemap will help you to identify folders that may hold a lot of content or have a special structure, either holding lots of files or very large files. Run the visualization code below and note the way the treemap encoded both of these metrics visually.

Can you identify the folders that contain the most data in terms of bytes? Which folders contain a bunch of smaller files?

Exercise: Analyzing Folder Structures¶

People organize files in a variety of ways. Sometimes their folder and sub-folder names are entirely indicative of the content. In other cases, folder names and structures are constrained by the technical considerations of the software tools that either produce or consume the files. As you look at the distribution of files, by area, and bytes, by color, across the top two folder levels what can you see?

What folder contained the largest files?

Why are there many folders called "www.bop.gov"?

Which folders are named to reflect a person's choices and which seem to be named due to a technical consideration?

Another View of Folder Structures¶

As mentioned at the start of this notebook, sometimes folder structures are part of a production workflow for a particular tool is discipline, such as GIS or audiovisual productions, or websites. One way to uncover such structures in larger accessions is to determine the most common folder and file names. Usually folders that are patterned by a workflow will reuse the same folder names or file names. These folder and filename "slots" determine the purpose of the folder or file within the larger workflow.

In the code below we take the "name" column from our data frame and convert the individual names into a shared set of unique categories (.astype("category")). Instead of separate text values, the names are now coded as category numbers. So each occurrence of the same name will have the same code. Pandas can then add up the occurrences of each code and give us value counts. (.value_counts()) Finally, we call the head() function to get just the first 25 most frequent names.

As we look at this list of the most frequent file names in this accession, we can clearly see that the most common file name is index.html, a web page file, with 45 occurances. Next there are some GIF files, usually these are web graphics. Then we have some files called "new.txt", "new.lst", and "readme.txt". It is clues like these that make digital appraisal somewhat of a detective story. You might dive deeper into these files, by reading the files themselves or by researching the file names on the Internet to see what workflows may have produced them.

TODO Differentiate Older and More Recent Folders¶

Depending upon the operating system, files will have a modified date and/or a created date. Usually the modified date is the more significant of these, because it indicates the last edit to a file and may indicate when it was completed or no longer relevant. When transferring files to our own storage we have to be careful to either capture or preserve these dates as metadata.

NOTE: Since this notebook is copied from a Git repository, the sample files included here do not have their original modified date. Since we cannot provide accurately dated files via Git, we are providing some alternative inventory CSV files that were prepared separately.

TODO: color by average timestamp TODO: find samples or supply an inventory file better good timestamps..

Dive into Folders¶

If your initial inventory reveals a sub-folder that is of particular interest, you can run the same visualizations against a specific subfolder. Let's pick a sub-folder and show the mimetype distribution of that subfolder by itself.

The first step is to subset our inventory dataframe, to include only the desired sub-folder.

Next we use the same treemap code as our prior mimetype visualization, but supply the new subfolder dataframe.

Exercise: Visualize a Sub-Folder¶

Pick a different top-level folder and then run a visualization that is limited to that folder. First subset the dataframe as shown above and then use that dataframe to run one of the prior visualizations.

Looking at a Larger Deposit Inventory¶

A second and larger set of sample files was inventoried previous and those files are not included with this notebook. However, the inventory file is supplied and you can use it to visual a larger deposit. Below we will load this inventory and visual it with the MIME Type chart.

You can see above that this inventory describes 10,413 files, a more challenging quantity for manual appraisal. Now let's run our visualization on this data.

Visualize Larger Deposit Folders¶

Frequent File Names¶

Let's reuse our code from the previous example to look over the common file names.