All rights reserved. ©2016 Board of Trustees of the University of Illinois.
This page is a resource for the talk that I gave at the SuperComputing Conference 2016. I will leave this page up through at least the first half of 2017.
I created two pieces of software for Blue Waters users that we are in the process of publishing open-source so that the wider community can use them.
The first package is specifically for Cray systems that use Environment Modules to control their software stack. We implemented several major imporovements in the interface to module and added some Cray-specific tweaks. This package makes the module interface much more fluid to use and adds the very important function of a dynamic prompt. The package is called "Module Improvements".
The second piece of code is a tool called "parfu" that archives directory trees of files into a single archive file and then unarchives them. Its general functionality is very similar to tar, although its archive files are not (yet) tar-compatible. Unlike tar or any of its bretheren (ptar, pgiz, among others) parfu is a full MPI program and scales and goes faster as you add more ranks. This tool will be useful for users and systems with workflows that create directory trees with many (millions) of small files and then need to archive them to tape. Storing one archive file of all the mall files, rather than the small files themselves, greatly improves the efficiency of storage. Parfu is fast enough that such archiving can be done faste enough to be part of the production workflow.
Github pages:
As of November 15, 2016, Module Improvements has been in production on Blue Waters for over a year. There is a version with one additional feature that's under test now. When that's tested, we will write the installers and release the code to the world. The hope is that will be in December 2016.
Parfu: As of May 2017, we understand how to buffer data to make the writes to disk faster. We also have code under way that will make the files tar-compatible but keep the very fast and parallel nature of the core parfu application. We hope to release the new (faster and tar-compatible) version in the summer of 2017.
When releases happen, announcements will be made to this page. If you're interested in hearing about updates, please send me a email at csteffen@ncsa.illinois.edu and put either "parfu" or "modimp" in the subject line.
All rights reserved. ©2016 Board of Trustees of the University of Illinois.