Cold Storage In Hsm For Raw Video And Images

THL Toolbox > Developers' Zone > Servers & Storage

Cold Storage in HSM of Original Images & Raw Digital Video

Contributor(s): David Germano

NOTE: as of 12 July 2013, blue.unix is no longer accessible. Access cold storage through another server, such as dev.thlib.org/net/coldstorage

We have 5 terabytes of space in the HSM system (Hierarchical Storage Management) to store original photos (Tiff, Raw, and Jpeg), and captured raw digital video. Our Media Management System (MMS) as part of the upload of media process stores the original image and raw dv to the HSM storage with the appropriate ID. Ideally, the RoR MMS would then store the link to that data for easy retrieval, even if that link is not publicly made available.

I have read external link: http://itc.virginia.edu/research/hsm.html. understand that is involves storage of data on tape, but that a tape robot allows for automatic retrieval and storage of tapes. I understand that recalling a file may take several minutes to be copied to disk. I understand that retrieving multiple files at one time can be extremely slow. I understand the absolute maximum size of any one file is 2 GB. For individual images, this is no problem since they are typically under 20 MB, and even special scans are under 100 MB. For audio-video, these would typically be digitized 60 minutes tapes of high definition video, which I understand to be about 13 GB. Thus it seems that the limit of 20 GB would work for us. One issue may be slide shows, but I would think slide shows would typically be 20 images and less, which should poseno problem.

As for frequency of retrieval, in general it will look like this:

  1. Someone may request the original of a single image to use in a publication, or perhaps internally to modify for specific uses in THL. There is no way to pre-group images together for such retrieval since typically requests for multiple images are drawn from areas of the image collections. To date, these have not been very frequent, though with the new launch of a MMS and THL they will likely more frequent than the past.
  2. Someone may return to a digitized footage to recompress or reedit the footage. However we can minimize such requests by uploading into cold storage first as an entire tape, and then also as individual edited pieces. Otherwise staff can use local hard drives for manipulation and only retrieve from cold storage when months later new work is required after local copies have been deleted.

Your HSM access has been setup. If you log into a server such as dev.thlib.org (NOTE: as of 12 July 2013, blue.unix is not longer accessible), then you can access the directory by typing, at the "$" system prompt, the command: cd /net/hsm/dfg9w and press enter and you'll switch to your HSM directory (the directory path /net/coldstorage might work also).

In addition, I've created a symbolic link (like a Windows or Macintosh alias) in your blue.unix home directory to the HSM directory, it's called "HSM". So from the "$" prompt, type the command: cd HSM and press enter and you'll switch to your HSM directory. Or you can double-click on that name in your Home Directory folder window.

We also set it up so the group THDL has read/write/execute permissions to this directory (/net/hsm/dfg9w/mms_coldstorage)

Members of this group can access this directory from blue.unix or their Windows or Macintosh computers the same way you do.

Both you and they use your blue.unix user-id and password to map the HSM drive. In Windows, you can use the Home Directory login program to connect to the HSM. In the Home Directory Service Login Window, in the "Server Name" box, put:
hsmdfg9w In the Username field put your blue.unix computing id (e.g., mst3k) and your blue.uxix password in the Password box.

  • PLEASE NOTE: THDL group members should use their own individual blue.unix computing id and passwords to connect/mount your HSM directory.

For Macintosh users, we're trying to resolve a cleartext password issue, so for right now, the guaranteed method of connection is to use FUGU or other secure ftp program. Again, one connects to blue.unix with his or her own individual computing id and password and then changes directory to your HSM directory (/net/hsm/dfg9w). In FUGU, you would click on the "Go To" icon, and in the "Go To:" box type /net/hsm/dfg9w and make sure "Remotely" is highlighted and click the "Go" button.

If you or any of your THDL group has difficulty accessing the HSM, or has other questions or problems with it, please let me know. You can use our Web help form at: external link: http://www.lib.virginia.edu/science/rescomp/help to report problems.

As to limits on transfers and file sizes. Apparently our web page on the HSM has not kept up with changes to the capacity. The file cache for the HSM is now 500GB in size, so transfers are not limited to 20GB a day. Also the 2 GB file size limit only applies to files being transferred from blue.unix, as that's an AIX file size limit. If you FTP or transfer files from elsewhere, e.g., Windows or Macintosh, then you're limited by their file size capacity.

You can mount the HSM folder read/write from Windows, Unix, or Macintosh. It should be possible to mount it with a consistent name (handle) and automate the writing.

  1. Open up a secure FTP client or a SCP client, such as FileZilla or FUGU (Mac) or Secure Fx or WINSCP.
  2. When it asks you where you want to connect, you type in blue.unix.virginia.edu. You can read about Blue Unix at external link: http://www.itc.virginia.edu/desktop/unix/cluster/home.html.
  3. Give your UVa login and password.
  4. Then have to find an option like “Go to” or “Change directory” and specify the folder /net/hsm/dfg9w (might have to use /net/coldstorage) to go to it directly.
  5. You should then see the folders. One is MMS Cold Storage. In that is a folder called “Movies”. Put the raw DV there. Please note that the speed can be variable.
  6. For file name, use the file name of the tape record for now.

Downloading Image Files from Cold Storage Using secure FTP software such as FileZilla

To locate image files:

  1. Take the MMS ID, such as 62059
  2. Add leading zeros to form an 8-digit number, such as 00062059
  3. The first 4 digits are the folder name in /images; in this case, 0006
  4. The last 4 digits are the file name, in this case 2059.tif
  5. Launch FileZilla and log into a server such as dev.thlib.org (AS OF 12 July 2013, blue.unix.virginia.edu is not accessible)
  6. Pull down the Transfer menu and select Manual Transfer (you can also use the keystroke command-m)
  7. Dialog box will open. At top of dialog box, set Transfer direction: download
  8. Local file: browse to the folder on your computer. Then at the end of that directory path, add the filename of the file you want to download; in this case, /2059.tif
  9. Under Remote file:
    1. Remote path: enter the directory path in cold storage, such as /net/hsm/dfg9w/mms_coldstorage/images/0006 (might need to use directory path /net/coldstorage in place of /net/hsm/dfg9w/mms_coldstorage)
    2. File: enter the file name, such as 2059.tif
  10. At bottom of window, click the box to the left of "Start transfer immediately"
  11. Press OK. File should download to the folder you specified on your computer

Provided for unrestricted use by the external link: Tibetan and Himalayan Library