Kangyur-tengyur Tiff Conversion Instructions

Tibetan Canons > Kangyur-Tengyur Scanning > Kangyur-Tengyur Tiff Conversion Instructions

Kangyur-Tengyur Tiff Conversion Instructions

Contributor(s): Than Grove, Chris Bell, Steve Weinberger

Instructions for Processing Page Images for Dege Canon

These instructions were devised for the Degé Kangyur and Tengyur scans. Because of the difference between Kangyur and Tengyur scans, there are some different conversion instructions for Photoshop, but the basic principle is the same. For the Dege Kangyur and Tengyur the basic conversion instructions have been recorded and only need slight modification to work on a particular computer. These are included in the initial download for setting up the conversion. For other editions, these general instructions apply. However, the cropping, resizing, and tone adjustment may not be exactly the same, and one will need to create unique Photoshop actions for these other editions.

Mac: The Basic Process of Conversion

Copy the volume of files to be processed to your desktop
Download the appropriate Photoshop Droplet to your desktop
1. Drag the Droplet itself to the desktop
2. Make sure that the folder THL Photoshop Droplet Routines is on your desktop; leave the empty sub-folder called "_final edits" in it since this is where the processed PNG files will be
Rename the volume of files to be processed using one of the two processes described below.
select all the files to be processed and drag them onto the Droplet
Photoshop will launch and will begin the automated processing of the images
When all the images are processed, go into the folder on your desktop THL Photoshop Droplet Routines > _final edits. All the processed PNG files should be there. Open a few of the files at random to make sure they have content.
Upload the PNG files to the server(dev or one of the sds servers), in the folder: /usr/local/projects/thlib-texts/static/kt/d/v### (where ###= the number of the volume) NOTE: to get the projects link, click on the local folder and you will see a symlink for projects
Zip the PNG files into a file named kt-d-v###-png.zip and upload to collab Tibetan Literature Resources > Kangyur-Tengyur Scans > LOC Dege Kangyur Scans > Processed PNG files > Volume ###. To copy the files:
1. download and install Cyberduck: http://www.apple.com/downloads/macosx/internet_utilities/cyberduck.html. After you double-click on the file you downloaded, you will want to drag the application into your Applications folder.
2. Launch Cyberduck and paste the site WebDAV URL (https://collab.itc.virginia.edu/dav/6a0a05fe-de5e-40f7-80f2-1ccfa2ab7d8e) into the Quick Connect field and press Return (Enter).
3. If prompted, choose WebDAV (HTTPS) as the protocol to use.
4. Enter your UVaCollab username and password and click Login.
5. Your UVaCollab site Resources will display in the Cyberduck window. You can now drag and drop files into this window and delete files by dragging them to the Trash.
6. Save this as a Favorite (or Library or whatever the word is; I can't remember and I don't have a Mac in front of me) with the name Tibetan Lit Resources so you don't have to set it up each time; you just click the word (favorite or library or whatever the word is) near the top left of the screen and in the pane below it will display a disk named Tibetan Lit Resources. When you double-click this it will open collab Tibetan Literature Resources.
Upload the tiff files to cold storage

Windows: Basic Process of Conversion

Set up conversion folder and Photoshop Action
Download page images and place them in the In folder
Rename the converted images using the free software Faststone ImageViewer at http://www.faststone.org/ (if this is not available for some reason, use the renaming script included in the download).
Run the Photoshop batch process on the In folder
Upload them to the server (dev or one of the sds servers)at: /usr/local/projects/thlib-texts/static/kt/d/v### (NOTE: to get the projects link, click on the local folder and you will see a symlink for projects)

I. Preliminaries

Download the PageImageProcessing.zip file and unpack its contents where desired on a computer that has Photoshop installed on it. It will unpack a folder called “ImageProcessing,” though you can rename it whatever you want if you choose. This folder will contain an “in” and an “out” folder.
Delete “place-holder” files in any of the unpacked folders
Load the Photoshop Action
1. Open Photoshop
2. Under the Window menu, make sure that the “Actions” window is checked.
3. From the Actions window, open the drop down menu by clicking on the right-pointing arrow just below the close button (X).
4. Choose the Load Actions option.
5. Navigate to the Dege Page Img Conversion.atn in the unzip folder from the downloaded setup .zip.
6. Press the Load button. A folder called “Dege Page Img Conversion” should appear in the Actions window. When that is opened, it should contain two sets of actions: ConvertKangyurScans and ConvertTengyurScans.
Customize the Photoshop Action
1. Open the appropriate Action in the Action Window. This is either ConvertKangyurScans or ConvertTengyurScans and make sure the various parts of the actions are showing by clicking on the arrow next to the name.
2. Each sub-action under the action has a check next to it. Uncheck the “Open” action (the first action) and the “Close” action (last action).
3. For the ConvertKangyurScans Action Only, re-record the Save as Tiff step so that the folder location is the Tiff folder on your computer:
  1. Delete the Save As Tiff step.
  2. Open an original scan (either JPG or raw file)
  3. Highlight the first step of the action, “Open”
  4. Press the record red circle at the bottom of the Action panel.
  5. Save the original scan as a TIFF (.tif) file in the Tiff folder
  6. Press the stop square at the bottom of the Action panel.
4. Delete the “Export” action (2nd to last) by clicking on it and dragging it to the trash can in the lower right corner of the Action window.
5. Open one of the original TIFF images and run the action on it by clicking the Run right-pointing arrow at the bottom right of the Actions window. This will resize the image but not save or close it.
6. Highlight the “Levels” action and press the record button (circle) in the lower right corner.
7. From the file menu, choose “Save for Web”
8. Under the label “Settings” in the resulting window, choose the PNG-8 option. For LOC Kangyur, choose “Grayscale”, “No dither”, and set “Colors” to 8. For Tengyur scans, choose “Web”, “No dither”, and set “Colors” to 10.
9. Press Save and locate the “out” folder and save the image in that folder without changing the name of the file.
10. Stop recording the macro by clicking on the square in the bottom right corner of the Action window.
11. Close the test image without saving it.

II. Conversion Basics

Locate the original files for one volume. If these files are in Collab, download them and put them in the In folder. If they are on a hard-drive, then connect the hard-drive to the computer.
Rename the original files to match the standard name structure, kt-d-v###-p####.ext, where .ext is the lowercase extension for the file, usually either .jpg or .png. For Mac this should be done with Automator, using the instructions immediately below; it can also be done with the PERL script using the instructions below. For Windows this should be done with Faststone ImageViewer, using the instructions below; it can also be done by following the instructions for using the Perl script below but this is more complicated.

Renaming files on a Mac: use Mac Automator (note: when you use this for the first time, create a new folder and copy 5 files into it, and run the file renamer as a test to make sure it works and does not overwrite files before you batch rename the entire volume. After you successfully test, delete the test files from the Output folder):

click the arrow next to Library. From the list on the right, drag the following into the far righthand pane: Get Specified Finder Items; Copy Finder Items; Rename Finder Items. Then configure each of these as follows:
Get Specified Finder Items: click the Add button, navigate into the folder with the files you want to rename, and select all the files you want to rename
Copy Finder Items: create a folder that you want to save the renamed files to (such as "Output"), and then select this folder using the pulldown menu
Rename Finder Item Names: in the pulldown menu at the upper left choose “Add text” (the name of this window may or may not change to "Add Text to Finder Item Names"); in the “Add” field, enter the text to be prepended to each file name; in the pulldown menu to the right of that, choose “before name” The text that goes in the Add field is: kt-d-v###-p
Hit the Run button at the top of the Window
You can then save this workflow so you can use it again.

Renaming Files on a Windows machine: use Faststone Image Viewer

Download and install the free software Faststone ImageViewer at http://www.faststone.org/
Launch the application
Pull down the Tools menu and point to "Open Batch Convert / Rename Dialog"
Click the "Batch Rename" tab
Above the box on the left, click the button with three dots and navigate to the folder of files you want to rename (I am using volume 039 as an example); click OK.
1. NOTE: be sure to do a test of this process the first time you use it. Create a folder named "test" and copy 5 files into it.
All the image files in that folder will now appear in the box on the left. Click the button "Add All" that is between the box on the left and the box on the right.
All the files will now appear in the box on the right.
On the right, below the box in which all the image files just appeared, click the radio button "Use Filename Template"
Click in the text to the right of "Template" and enter this text: kt-d-v039-p#### (note: this example is for volume 039; you will need to change the number you enter according to the volume number you are renaming)
Make sure the number in the box the right of this is set at "1"
Make sure the Ext. Case is set at "No Change"
Click the "Rename" button on the bottom right.
Check the files to make sure they were renamed correctly

Instructions for renaming files using the PERL script, for Windows and Mac (note: it is easier to use the process described above):

This renaming is done using a PERL script included with the converter, called renamePgImgs.pl. The files to be renamed have to be in the "in" folder which is in the ImageProcessing folder (to locate an external hard drive on a Mac, open Terminal and type "cd /Volumes"; then type "ls" and you will see the name of the hard drive).
1. To rename the files with the PERL script on Windows, it is recommended to open a DOS window. Do this by clicking on the “dos.bat” file. In Mac OS X, one can do this by opening Terminal, which is located in the Utilities folder of your Applications folder (or, to use the Unix convention, /Applications/Utilities). Double-click the Terminal icon. Then navigate to the ImageProcessing folder.
2. Run the PERL renaming script by typing renamePgImgs.pl (for Windows) and perl renamePgImgs.pl (for Mac).
3. It will ask you the volume number to use for the file names, enter the 3-digit volume number with leading-zeros as necessary. Press enter. This will rename the files in the in folder as kt-d-v###-p####.ext, assuming that the last 4 digits of the existing filename is the sequential page number, starting with 0001, 0002, on up.
Run the Photoshop batch conversion on the
1. Under the File menu, choose Automate, Batch. The “Batch” dialog will show up.
2. Choose the “Set” called "Dege Page Img Conversion
3. Choose the appropriate action, such as “Dege Kangyur Pg Conv”.
4. Under the “Source” drop-down choose the “Folder” option.
5. Press the Choose button and navigate to the folder that contains the Tiff files you want to convert. These could be in the “in” folder or could be on an external hard-drive.
6. Click the checkbox “Override Action “Open” Commands”
7. Under the “Destination drop-down choose the “Folder” option.
8. Click the Choose… button and navigate to the “…ImageProcessingout” folder and choose it.
9. Uncheck the checkbox “Override Action “Save As” Commands
10. Under “Errors” drop-down, choose “Log Errors to File”. Click the SaveAs button and choose an appropriate location and file name for the log file that will be created, such as “…ImageProcessingerrorlog.txt”
11. Press Ok to begin the batch conversion.
Wait for the batch process to finish. This could take a while depending on how many images are being converted.

Note: you need to enter in the JIRA status issue the number of images for sides before side 1a (which is the blank front side of the first side with text on it, which is 1b). Then the XML volume catalog record for that volume needs to be modified as follows:

add a new field to the extent decl. An example from kt-d-v001-bib.xml is:

&lt;extentdecl type="sides"&gt;
&#x20;&#x20;&#x20;&lt;extent class="sides-before-1a"&gt;1&lt;/extent&gt;
&#x20;&#x20;&#x20;&lt;extent class="blank"&gt;1&lt;/extent&gt;
&#x20;&#x20;&#x20;&lt;extent class="differential"&gt;-0.571&lt;/extent&gt;
&#x20;&#x20;&#x20;&lt;extent class="total"&gt;619.143&lt;/extent&gt;
&lt;/extentdecl&gt;

The first extent "sides-before-1a" should have the number of images prior to the 1a one. Thus, the images do not need to be renamed.

Converting Large JPG Images Created while Photographing LOC Kangyur into Smaller JPG Images

Mac or Windows, Using Photoshop

Go to the JIRA issue THLLIT-154 (http://jira.shanti.virginia.edu/jira/browse/THLLIT-154) for this task. Edit the description and in the table entry for the volume you are going to work on, enter your initials in the "Processed by (Initials)" column; in the "Date Processed" column, enter "Working"
Rename the images using the procedure listed above appropriate for your computer platform (Mac or Windows)
Launch Photoshop
Pull down the File menu and point to Scripts > Image Processor
In section 1 at the top of the dialog box, click "Select Folder" and navigate to the folder that the files are in; select "choose"
In section 2, click the radio button to the left of "Select Folder" and create a new folder named "kt-d-v###-processed JPGs"
On the next screen, choose the folder you just created
In section 3 (File Type)
1. Save as=JPG
2. Click the box next to "Resize to Fit"
3. In the W box, enter 2272
4. In the L box, enter 1515
Click "Save" button
1. In the Save As field enter Processing LOC Kang JPGs into smaller JPG files
2. Click the save button and choose a folder (not the desktop)
Now in the Photoshop Image Processor dialog box, click "Load" and navigate to the file Processing LOC Kang JPGs into smaller JPG files
Click "Run" and it will process the images

Creating a PDF with the Processed JPGs

Launch Adobe Acrobat Professional
Pull down the File menu and point to Create PDF > From Multiple Files
Click the Add Files icon at the top left; navigate to the files you want to combine and select them
1. For each volume, combine files 1-400 into a PDF and files 401-end into a separate PDF
Keep the default settings at the bottom and click the "Next" button at the bottom
Make sure the "Merge files into a single PDF" is selected
Click "Create" and it will create the PDF (should take between 5 and 10 minutes).
After it creates the PDF, save the file with this filename: kt-d-v###-p0001-0400
1. Be sure to set the page numbers according to the images that are actually in the file. If the PDF is of volume 37, pages 401-793, then name it kt-d-v037-p0401-0793
Upload the file to Collab Tibetan Literature > Resources > LOC Dege Kangyur Scans > Processed PDF Files
Update the status in the JIRA issue: in the Date Processed column, "working" with the date in the format 2010-mm-dd

When you next launch Photoshop to start working again, pull down the File menu and point to Scripts > Image Processor
Click the "Load" button and navigate to the file you created, Processing LOC Kang JPGs into smaller JPG files
In section 1, select the folder the files you want to process are in
In section 2, select the folder you want the processed files to go to
Click the Run button

III. Post Conversion

For computers using the latest version of Photoshop, PNG files are automatically compressed. For older versions of Photoshop, such as Photoshop 7, it is possible to compress PNGs further after conversion. They should be compressed if the compression significantly reduces the file size. Whether compressed or not, the converted images need to be uploaded to THL’s static site and the renamed originals need to be put in the tape-backup archive.

In Windows, you can run a compression utility called PNGOUT. This is done by typing compressPngs.bat. This will create a folder called compressed and place the final compressed files there. (Mac users can also try it. Check http://www.jonof.id.au/kenutils for the installation files. You will have to write the script yourself.) This is NOT a required step and probably will not be necessary if you are using the latest version of Photoshop. This will make a “compressed” folder and put the compressed version of PNGs in there. (JPGs will not be compressed by this utility.)
Upload the final files in the “compressed” folder (if you compressed them) or in the “out” folder (if you did not compress) to blue.unix.virginia.edu: /net/quandu/thdl-texts/kt/d/v### without .zipping them up.
Upload the renamed originals in the “in” folder to the tape-backup archive (instructions to be determined).

Notes

These notes are saved from previous versions of the page and describe how to record your own action and use it for batch conversion

to determine what processing needs to be done to the images--cropping, resizing, darkening, contrast, etc.--and then in Photoshop recording a macro to do all these things. This macro is then applied to all the images for one volume en masse using the Batch command. When determining the amount of contrast, darkening, etc. that should be applied to images. It is best to experiment with images at the beginning, middle, and end of a volume as the amount of ink used in the printing tends to wane further on in a volume.

Recording Conversion Actions in Photoshop

Under the Window menu, choose the “Actions”
Click on the folder icon () at the bottom of the Action palette to create a new folder.
Name this folder something appropriate, such as “Page Img Conversion”
With the new folder highlighted, click on the new action icon () at the bottom of the Action palette. The “New Action” dialog will appear.
Set the name of the new action to “Dege Kangyur Pg Conv”, make sure the “Set:” value is the same as the folder name, “Page Img Conversion”, and then click on the “Record” button.
You are now recording a macro. Do whichever of the following actions are appropriate to make the desired form of the image:
1. Open Image File: Open a sample raw TIFF page image from the “in” folder.
2. Rotate Image: Determine if the images need to be rotated and rotate the appropriate direction by choosing Image menu, and then Rotate Canvas sub menu, and then the appropriate option.
3. Crop: Determine the amount of minimum amount of whitespace on the left side of the image. Under the Image menu, choose Canvas Size. Choose the right middle area so that it crops the left side. Then set the width to be shorter by the minimum amount of white space. For instance if the width is 15500 and there is at least 1500 pxs of white space to the left of the image, enter a width of 14000, making sure to choose the option where it will crop from the left side. This will crop the border and whitespace around the image. It will ask you if you want to do this. Click on the “Proceed” button. Repeat the procedure for cropping extra whitespace on the right side, and/or top and bottom. Note: be careful about cropping images as the scans will vary from page to page. Always crop only consistently extra whitespace.
4. Resize: Under the Image menu, choose Image size. Set the Width to “1800” and click OK. This will resize the image, scaling the height to match the width. (At this point the image will appear tiny in Photoshop. You can zoom in on the image by pressing Ctrl + any number of times without this being recorded in the macro.)
5. Lighten: This step needs to be finalized Adjust the contrast and clarity of the scans. Choose the Image menu, Adjustments, Autocontrast. Or you can set the contrast by hand. Go to Image menu. Choose Brightness/Contrast, and set the Contrast as desired (Suggested value between 30 to 50, but some instances may require more contrast. Pick a value that seems a good average to cover all images in the folder.)
6. Save: From the File menu, choose, Save For Web option. Under the "Settings:" label, there is a drop down menu from which you should choose "PNG-8" and then click the Save button.
7. Stop Recording Macro: At the bottom of the Actions palette, click the stop button () to stop recording the macro.
For each different kind of output file or if there are differences in the input files, repeat steps 5 through 7 (mutatis mutandis) to create other conversion macros (which are called “Actions”, such as a “Rotate Clockwise” action, etc.). These can all be created in the “Page Img Conversion” folder.

Note: Should the images in general require other processing. The appropriate steps can be done anywhere between opening and saving the file. Remember each step will be applied to each file in the batch folder. So, only steps that can be generally applied to all files should be done. Furthermore, these actions once created will be automatically saved within Photoshop. This was developed on PS 7.0, which does not have a way to export and import actions, though this may be possible in subsequent releases of the program, in which case this only needs to be done once and shared. Also, in developing this process, there were situations where actions no longer worked when called by a batch process. I believe this was due to the folder save to in step 7f was moved or deleted. If this happens, make sure the “jpgs” folder is in your “out” folder. Sometimes if the “Open” or “Save” parts of the macro need to be rerecorded, you can highlight the action before it and press the record button ( Action Palette Record Button ), perform the appropriate actions, then stop. Old actions or steps can be thrown away by dragging them into the trash can ( Action Palette Trash Can ) at the bottom of the Action palette. I believe in later versions of Photoshop it is possible to export actions so they can be saved and loaded into another machine.

Updating Export Steps in an Action Script

Sometimes a script created on another computer will not work on a different computer because there is an Export step that specifies a folder not on the converting computer. This is especially true when running a script created on a Mac on a Windows machine. To fix this do the following, after loading the script into Photoshop:

In the Action panel, open the script by clicking on the arrow next to it so you can see all the steps
Each step should have a check mark next to it. Uncheck the "Export" and "Close" step.
Open a test image
Run the action on the image. It should do all the converting without saving or closing the file.
Click on the (unchecked) "Export" step in the list of the action's steps. This will open up the export window.
Navigate to a folder on your machine and click Save. (If you are going to "Override" the save as action in the batch process, it does not matter which folder you choose here. Any folder will do.)
Recheck the "Export" and "Close" steps.
Do a batch conversion of images as described above.