Computational Photography (1051-753)

Camera arrays and plenoptic cameras are used to capture light fields for refocusing, 3D displays, and many other applications.	Lytro is a recent cool light-weight camera that allows you change focus after taking pictures and many more.	The Nokia N900, a Linux-based camera phone with a 5-megapixel camera, focusable lens, WiFi, and touchscreen.	Computational lighting has been widely used in movie industry to capture live performance of actors and render in virtual environment.	Global illumination, such as the inter-reflection between the eggs, can now be easily separated with computational illumination.

Course Information

Class: 4. Credit: 4

Instructor: Jinwei Gu
Email: jwgu@cis.rit.edu
Office: CAR(76)-3262
Phone: 585-475-6783
Office Hours: Monday 3:50pm-4:50pm (or by appointment).

TA: TBA
Email: TBA@rit.edu
Office: TBA
Office Hour: TBA

Description

Computational photography is an emerging field that aims to overcome the limitations of conventional digital imaging and display devices by using computational techniques and novel coded optical devices and sensors to perform more efficient and accurate measurement as well as produce more compelling and meaningful visualizations of the world around us. It is a convergence of many areas, such as computer vision, computer graphics, image processing, photography, and so on.

In this course, we will study many interesting, recent techniques for capturing, modeling, and displaying of complex appearance phenomena. Students will implement some of these techniques. We will cover topics such as computational sensors with assorted pixel, mobile camera control, light field capture and rendering, computational flash photography, computational illumination for appearance acquisition and 3D reconstruction, reflectance transformation imaging, light transport analysis, novel displays and printing techniques.

The course will consist of six required programming homework, two optional programming homework with bonus points, and a presentation and paper review about one relevant paper. There is no midterm or final exam. We will provide a Nokia N900 cell phone camera with open source SDK as a test-bed for this course.

Compared to the previous offering of this course, this year we make several major changes:

• Each week we have two class sessions (Monday and Wednesday).
• There is no final project.
• We add more contents on 3D computer vision, such as camera projective modeling, calibration, stereo matching, structure from motion, and structured light.
• We emphasize more on programming homework.
• Students are expected to take pictures, write code to implement algorithms, and write reports.

Prerequisite

Basic knowledge of radiometry, image processing, and linear algebra. Programming skills are mainly Matlab and some C/C++ programming (skeleton code will be provided in some cases). If you are not sure whether you can take the course, please send me email or talk to me!

Topics

• Image Formation, Camera Model
• Computational Sensor, Assorted Pixel
• HDR Imaging
• Light Field Capture and Rendering
• Computational Camera
• Computational Flash Photography
• Camera Projective Modeling and Calibration
• Stereo Matching
• Structure from Motion
• Structured Lighting Depth Recovery
• Photometric Stereo
• BRDF Acquisition
• Light Transport Analysis
• Image Relighting
• Light Multiplexing
• Novel Displays and Printing
• Compressive Sensing
• ...

Course Format

• Lectures: The instructor will give the lecture each week, covering a background introduction of the topic and the key points/skills for each topic.
• Paper Review and Presentation: We will have a mini conference in the last class. Each student will present a relevant paper (usually a recent publication in the referred conference/journal) ( 15 min + 5 min discussion). The presenter will also fill out a review form for the selected paper.
• Homework: There will be SIX required programming homework. Plus, there will bet TWO optional programming homework.

Grading

• Homework (required HW: 72%=6*12%. optional HW: 24%=2*12%)
• Paper Review/Presentation (20%=5%+15%)
• Class Attendance (8%)

The bonus points from the two optional homework will be added to your score upto 100% (i.e., the maximum score you can get will not be greater than 100%).

No late submission will be accepted except in the case of genuine documented emergency.

Texts

• Forsyth and Ponce. Computer Vision: A Modern Approach. Pearson. 2002.
• Richard Szeliski. Computer Vision: Algorithms and Applications. Springer. 2010.
• Berthold K. P. Horn. Robot Vision. The MIT Press. 1986.
• R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision, Cambridge Press, Cambridge, UK, 2000.
• Rastislav Lukac. Computational Photography: Methods and Applications. CRC Press. 2010.

Homework

• HW0: Choose A Paper to Review and Present
• HW1: Camera Noise Measurement and Modeling
• HW2: HDR Imaging and Tone Mapping on FrankenCamera
• HW3: (Optional) Light Field Rendering
• HW4: Camera Geometric Calibration
• HW5: Stereo Matching
• HW6: Photometric Stereo
• HW7: Separation of Direct and Global Illumination
• HW8: (Optional) Polynomial Texture Mapping

Submissions

Tentative Schedule

Lecture Notes: Slides presented in class will be posted in the content area of myCourses. The tentative syllabus is subject to change (but not much) and update.

	Date	Topic	Date	Topic	Assignment
Week 1	9/3	Introduction	9/5	Radiometry Review	HW0 OUT
Week 2	9/10	Image Formation and Camera	9/12	Image Sensors and Noise	HW0 BACK HW1 OUT
Week 3	9/17	HDR Imaging	9/19	FrankenCamera	HW1 BACK HW2 OUT
Week 4	9/24	Light Field (I)	9/26	Light Field (II)	Optional HW3 OUT
Week 5	10/1	Camera Projective Geometry	10/3	Binocular Stereo	HW2 BACK Optional HW3 BACK HW4 OUT
Week 6	10/8	Stereo Matching	10/10	Structure From Motion	HW4 BACK HW5 OUT
Week 7	10/15	Illumination Multiplexing	10/17	Photometric Stereo	HW5 BACK HW6 OUT
Week 8	10/22	Structured Light Depth Recovery	10/24	Appearance Capture
Week 9	10/29	Light Transport Analysis	10/31	An Introduction to Compressive Sensing	HW6 BACK HW7 OUT
Week 10	11/5	Image Relighting	11/7	Mini Conference	HW7 BACK Optional HW8 OUT
Week 11	11/12	TBA	11/14	TBA	Optional HW8 BACK

Useful Links

Similar Courses in Other Universities:

• Computational Photography SIGGRAPH Course (Raskar & Tumblin)
• Computational Camera and Photography (Raskar, MIT)
• Digital and Computational Photography (Durand & Freeman, MIT)
• Computational Photography (Levoy & Wilburn, Stanford)
• Computational Photography (Belhumeur, Columbia)
• Computational Photography (Efros, CMU)
• Computational Photography (Essa, Georgia Tech)
• Computational Photography (Fergus, NYU)
• Computer Vision (Seitz, U of Washington)
• Computer Vision (Zhang, U of Wisconsin)
• Computer Vision (Snavely, Cornell)
• Introduction to Visual Computing (Kutulakos, U of Toronto)
• Online book, Computer Vision: Algorithms and Applications, by Richard Szeliski.

More Links

• What is Computational Camera, Shree Nayar, Columbia
• Columbia Projects, Shree Nayar, Peter Belhumeur
• MIT Projects, Fredo Durand, William Freeman, Edward Adelson, Antonio Torralba, Raskar Ramesh
• Stanford Projects, Marc Levoy and collaborators
• USC Projects, Paul Debevec and collaborators
• CMU Projects, Narasimhan, Efros
• Jack Tumblin's 'Questions' for the field
• Richard Szeliski's online Computer Vision Book
• Conferences: ICCP 2011, ICCP 2010, ICCP 2009, SIGGRAPH, SIGGRAPH Asia, CVPR, ICCV, ECCV, ...

Acknowledgement

Many of the course materials are modified from the excellent class notes of similar courses offered in other schools by Prof Yung-Yu Chung, Frédo Durand, Alexei Efros, William Freeman, Shree Nayar Peter Belhumeur Marc Levoy Noah Snavely Li Zhang Srinivasa Narasimhan, Steve Seitz, and Dr Richard Szeliski. The instructor is extremely thankful to the researchers for making their notes available online. Please feel free to use and modify any of the slides but acknowledge the original sources where appropriate.