Project 1

Overview

Given the digitized Prokudin-Gorskii glass plate images, which consist of three color channels of a scene (taken in the early 1900s), we want to produce a color image using the three black and white channels, representing the blue, green and red channels of the original scene that was photographed. This typically is done by taking the three channels, colorizing them appropriately, then aligning the three such that there are minimal visual artifacts in the final result.

Approach

Generally speaking, the approach to align the different channels was to set a reference channel (in this case blue), and align the other two individually with the blue channel.

In order to find the transformation vectors to align each channel, I needed a way to quantify how well a vector is able to align, and in this case I chose NCC (Normalized Cross Correlation), which essentially means to normalize the two channels, then take the dot product between the two flattened channel arrays.

With this metric, I searched all the possible vectors ([-15. -15] to [15, 15]) and took the vector with the max NCC. This method works well for smaller images where that window fits the range for the desired adjustment vectors, however it doesn’t for larger images.

For larger images, I opted for the image-pyramid method, which consisted of recursively downscaling until we could find a vector within the window that contains the correct adjustment vector (I manually selected the parameters used to define when to stop downscaling, and how big the window). Then, once we hit the base case, we recursively upscale the image, as well as the alignment vector we found from the previous recursive case, and adjust this alignment vector based on the window we defined.

Results

Emir.tif

Not sure what was going on in the bottom photo, but it seems that the red channel was shifted a bit too much. I eventually found some parameters that seemed to work well: 5% cropping of the edges, scale factor of 2, window size of [-15, 15], and base case image size of 500.

Summarized Output (reference for all photos except emir.tif):

data/church.tif      -> out/batch5/church_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (-8, 52)
GREEN adj: (-5, 0)

data/emir.tif        -> out/batch5/emir_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (16, 104)
GREEN adj: (7, -3)

data/harvesters.tif  -> out/batch5/harvesters_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (0, 96)
GREEN adj: (0, 112)

data/icon.tif        -> out/batch5/icon_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (24, 88)
GREEN adj: (16, 44)

data/lady.tif        -> out/batch5/lady_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (-16, 112)
GREEN adj: (-4, 56)

data/melons.tif      -> out/batch5/melons_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (0, 176)
GREEN adj: (0, 80)

data/onion_church.tif -> out/batch5/onion_church_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (8, 104)
GREEN adj: (20, 52)

data/sculpture.tif   -> out/batch5/sculpture_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (-16, 144)
GREEN adj: (-12, 32)

data/self_portrait.tif -> out/batch5/self_portrait_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (0, 128)
GREEN adj: (-4, 48)

data/three_generations.tif -> out/batch5/three_generations_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (8, 104)
GREEN adj: (4, 52)

data/train.tif       -> out/batch5/train_pyramid_ncc-nocrop_2-8-50.tif
RED adj: (0, 104)
GREEN adj: (-8, 112)

data/cathedral.jpg   -> out/batch5/cathedral_pyramid_ncc-nocrop_2-8-50.jpg
RED adj: (-1, 7)
GREEN adj: (-1, 1)

data/monastery.jpg   -> out/batch5/monastery_pyramid_ncc-nocrop_2-8-50.jpg
RED adj: (1, 9)
GREEN adj: (0, -6)

data/tobolsk.jpg     -> out/batch5/tobolsk_pyramid_ncc-nocrop_2-8-50.jpg
RED adj: (3, 6)
GREEN adj: (2, 3)

Full Raw Output:

data/church.tif      -> out/batch5/church_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9607, 3634)
RED adj:
h,w: 50 56 | window: 0 0 8
h,w: 100 113 | window: 0 2 8
h,w: 200 227 | window: 0 2 8
h,w: 400 454 | window: -2 6 8
h,w: 800 908 | window: -2 12 8
h,w: 1601 1817 | window: -4 26 8
h,w: 3202 3634 | window: -8 52 8
(-8, 52)
GREEN adj:
h,w: 50 56 | window: 0 0 8
h,w: 100 113 | window: 0 0 8
h,w: 200 227 | window: 0 0 8
h,w: 400 454 | window: 0 0 8
h,w: 800 908 | window: -2 0 8
h,w: 1601 1817 | window: -2 0 8
h,w: 3202 3634 | window: -6 0 8
(-5, 0)
data/emir.tif        -> out/batch5/emir_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9627, 3702)
RED adj:
h,w: 50 57 | window: 0 0 8
h,w: 100 115 | window: 0 4 8
h,w: 200 231 | window: 0 6 8
h,w: 401 462 | window: 2 14 8
h,w: 802 925 | window: 4 26 8
h,w: 1604 1851 | window: 8 52 8
h,w: 3209 3702 | window: 16 104 8
(16, 104)
GREEN adj:
h,w: 50 57 | window: 0 0 8
h,w: 100 115 | window: 0 0 8
h,w: 200 231 | window: 0 0 8
h,w: 401 462 | window: 0 0 8
h,w: 802 925 | window: 2 0 8
h,w: 1604 1851 | window: 4 -2 8
h,w: 3209 3702 | window: 8 -4 8
(7, -3)
data/harvesters.tif  -> out/batch5/harvesters_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9656, 3683)
RED adj:
h,w: 50 57 | window: 0 0 8
h,w: 100 115 | window: 0 2 8
h,w: 201 230 | window: 0 6 8
h,w: 402 460 | window: 0 12 8
h,w: 804 920 | window: 0 24 8
h,w: 1609 1841 | window: 0 48 8
h,w: 3218 3683 | window: 0 96 8
(0, 96)
GREEN adj:
h,w: 50 57 | window: 0 0 8
h,w: 100 115 | window: 0 2 8
h,w: 201 230 | window: 0 6 8
h,w: 402 460 | window: 0 14 8
h,w: 804 920 | window: 0 28 8
h,w: 1609 1841 | window: 0 56 8
h,w: 3218 3683 | window: 0 112 8
(0, 112)
data/icon.tif        -> out/batch5/icon_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9732, 3741)
RED adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 116 | window: 0 2 8
h,w: 202 233 | window: 0 6 8
h,w: 405 467 | window: 2 10 8
h,w: 811 935 | window: 6 22 8
h,w: 1622 1870 | window: 12 44 8
h,w: 3244 3741 | window: 24 88 8
(24, 88)
GREEN adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 116 | window: 0 2 8
h,w: 202 233 | window: 0 4 8
h,w: 405 467 | window: 2 6 8
h,w: 811 935 | window: 4 10 8
h,w: 1622 1870 | window: 8 22 8
h,w: 3244 3741 | window: 16 44 8
(16, 44)
data/lady.tif        -> out/batch5/lady_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9637, 3761)
RED adj:
h,w: 50 58 | window: 0 0 8
h,w: 100 117 | window: 0 4 8
h,w: 200 235 | window: 0 8 8
h,w: 401 470 | window: -2 14 8
h,w: 803 940 | window: -4 28 8
h,w: 1606 1880 | window: -8 56 8
h,w: 3212 3761 | window: -16 112 8
(-16, 112)
GREEN adj:
h,w: 50 58 | window: 0 0 8
h,w: 100 117 | window: 0 2 8
h,w: 200 235 | window: 0 4 8
h,w: 401 470 | window: 0 6 8
h,w: 803 940 | window: -2 14 8
h,w: 1606 1880 | window: -2 28 8
h,w: 3212 3761 | window: -4 56 8
(-4, 56)
data/melons.tif      -> out/batch5/melons_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9724, 3770)
RED adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 117 | window: 0 4 8
h,w: 202 235 | window: 0 10 8
h,w: 405 471 | window: 0 22 8
h,w: 810 942 | window: 0 44 8
h,w: 1620 1885 | window: 0 88 8
h,w: 3241 3770 | window: 0 176 8
(0, 176)
GREEN adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 117 | window: 0 2 8
h,w: 202 235 | window: 0 6 8
h,w: 405 471 | window: 0 10 8
h,w: 810 942 | window: 0 20 8
h,w: 1620 1885 | window: 0 40 8
h,w: 3241 3770 | window: 0 80 8
(0, 80)
data/onion_church.tif -> out/batch5/onion_church_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9646, 3781)
RED adj:
h,w: 50 59 | window: 0 0 8
h,w: 100 118 | window: 0 4 8
h,w: 200 236 | window: 0 6 8
h,w: 401 472 | window: 2 14 8
h,w: 803 945 | window: 2 26 8
h,w: 1607 1890 | window: 4 52 8
h,w: 3215 3781 | window: 8 104 8
(8, 104)
GREEN adj:
h,w: 50 59 | window: 0 0 8
h,w: 100 118 | window: 0 2 8
h,w: 200 236 | window: 0 4 8
h,w: 401 472 | window: 2 8 8
h,w: 803 945 | window: 4 14 8
h,w: 1607 1890 | window: 10 26 8
h,w: 3215 3781 | window: 20 52 8
(20, 52)
data/sculpture.tif   -> out/batch5/sculpture_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9773, 3800)
RED adj:
h,w: 50 59 | window: 0 0 8
h,w: 101 118 | window: 0 4 8
h,w: 203 237 | window: -2 8 8
h,w: 407 475 | window: -2 18 8
h,w: 814 950 | window: -4 36 8
h,w: 1628 1900 | window: -8 72 8
h,w: 3257 3800 | window: -16 144 8
(-16, 144)
GREEN adj:
h,w: 50 59 | window: 0 0 8
h,w: 101 118 | window: 0 0 8
h,w: 203 237 | window: 0 2 8
h,w: 407 475 | window: -2 4 8
h,w: 814 950 | window: -2 8 8
h,w: 1628 1900 | window: -6 16 8
h,w: 3257 3800 | window: -12 32 8
(-12, 32)
data/self_portrait.tif -> out/batch5/self_portrait_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9754, 3810)
RED adj:
h,w: 50 59 | window: 0 0 8
h,w: 101 119 | window: 0 4 8
h,w: 203 238 | window: 0 8 8
h,w: 406 476 | window: 0 16 8
h,w: 812 952 | window: 0 32 8
h,w: 1625 1905 | window: 0 64 8
h,w: 3251 3810 | window: 0 128 8
(0, 128)
GREEN adj:
h,w: 50 59 | window: 0 0 8
h,w: 101 119 | window: 0 2 8
h,w: 203 238 | window: 0 4 8
h,w: 406 476 | window: 0 6 8
h,w: 812 952 | window: 0 12 8
h,w: 1625 1905 | window: -2 24 8
h,w: 3251 3810 | window: -4 48 8
(-4, 48)
data/three_generations.tif -> out/batch5/three_generations_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9629, 3714)
RED adj:
h,w: 50 58 | window: 0 0 8
h,w: 100 116 | window: 0 4 8
h,w: 200 232 | window: 0 6 8
h,w: 401 464 | window: 0 14 8
h,w: 802 928 | window: 2 26 8
h,w: 1604 1857 | window: 4 52 8
h,w: 3209 3714 | window: 8 104 8
(8, 104)
GREEN adj:
h,w: 50 58 | window: 0 0 8
h,w: 100 116 | window: 0 2 8
h,w: 200 232 | window: 0 4 8
h,w: 401 464 | window: 0 6 8
h,w: 802 928 | window: 2 14 8
h,w: 1604 1857 | window: 2 26 8
h,w: 3209 3714 | window: 4 52 8
(4, 52)
data/train.tif       -> out/batch5/train_pyramid_ncc-nocrop_2-8-50.tif
h,w: (9715, 3741)
RED adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 116 | window: 0 2 8
h,w: 202 233 | window: 0 6 8
h,w: 404 467 | window: 0 12 8
h,w: 809 935 | window: 0 26 8
h,w: 1619 1870 | window: 0 52 8
h,w: 3238 3741 | window: 0 104 8
(0, 104)
GREEN adj:
h,w: 50 58 | window: 0 0 8
h,w: 101 116 | window: 0 4 8
h,w: 202 233 | window: 0 8 8
h,w: 404 467 | window: 0 14 8
h,w: 809 935 | window: -2 28 8
h,w: 1619 1870 | window: -4 56 8
h,w: 3238 3741 | window: -8 112 8
(-8, 112)
data/cathedral.jpg   -> out/batch5/cathedral_pyramid_ncc-nocrop_2-8-50.jpg
h,w: (1024, 390)
RED adj:
h,w: 85 97 | window: 0 0 8
h,w: 170 195 | window: 0 4 8
h,w: 341 390 | window: 0 8 8
(-1, 7)
GREEN adj:
h,w: 85 97 | window: 0 0 8
h,w: 170 195 | window: 0 0 8
h,w: 341 390 | window: 0 0 8
(-1, 1)
data/monastery.jpg   -> out/batch5/monastery_pyramid_ncc-nocrop_2-8-50.jpg
h,w: (1024, 391)
RED adj:
h,w: 85 97 | window: 0 0 8
h,w: 170 195 | window: 0 4 8
h,w: 341 391 | window: 0 8 8
(1, 9)
GREEN adj:
h,w: 85 97 | window: 0 0 8
h,w: 170 195 | window: 0 -2 8
h,w: 341 391 | window: 0 -6 8
(0, -6)
data/tobolsk.jpg     -> out/batch5/tobolsk_pyramid_ncc-nocrop_2-8-50.jpg
h,w: (1024, 396)
RED adj:
h,w: 85 99 | window: 0 0 8
h,w: 170 198 | window: 2 2 8
h,w: 341 396 | window: 2 6 8
(3, 6)
GREEN adj:
h,w: 85 99 | window: 0 0 8
h,w: 170 198 | window: 0 0 8
h,w: 341 396 | window: 2 2 8
(2, 3)