StorageMAP scanning 3.5 billion files at the University of Manchester

The UK’s University of Manchester is using Datadobi file mapping software to identify old files and move them into lower-cost archive storage.

The university has a stellar early involvement with computing. The first high-speed, entirely electronic memory was developed at Manchester University in 1947. The concept of virtual memory emerged from a team at the University of Manchester, with Ferranti involvement, and its Atlas computer in 1962. Virtual memory permitted a computer to use its storage capacity to switch rapidly among multiple programs or users and was a key requirement for timesharing.

Now it has a variety of IT systems supporting its teaching and research activities, amongst which is its Research Data System, This uses a Dell PowerScale, Isilon as was, clustered filer to store research files, with up to 15TB a day coming in. Last year the 5-year refresh date for the PowerScale, with its 10 PB capacity came due, and the University was looking at a costly upgrade to 20PB.

Realizing that a lot of the c3.5 billion files in the system were getting old, past their use-by-date as it were, it went looking for an unstructured data management tool to scan and analyse the metadata. It would allow them to tag files/folders with tags such as "cold", "retain-until-2040", "faculty", "school", and "classification." The tool would allow it to move data to a new, tape-based, cold storage platform.

Imagine that you are the IT admin team lead at the university and you have to check the status of 3.5 billion files to find the older, under-accessed ones, collect them together, and move them to a tape system. It’s an impossibility, and would take years to check them all.

The University settled on Datadobi’s StorageMAP product to automate the process. This can scan file metadata and do it in far, far less time than a system admin team.

Wayne Smith, Research Data Management Lead at the University, said: “The challenge was identifying which datasets among billions of files could be safely moved to archive storage. A manual approach would have required scripting through massive volumes of data, consuming significant staff time and introducing risk through human intervention. StorageMAP gives us the visibility and confidence to make these decisions efficiently, transforming how we manage our research data.” 

StorageMAP identifies ageing and unused datasets suitable for archiving so they can be removed from the primary storage. This will, Datadobi says, enable the University to make significant cost savings over the next five years by efficiently identifying and archiving ageing data, significantly reducing the need for costly primary storage expansion. Dell's loss is Datadobi's gain.