The New Jersey Institute of Technology's
Electronic Theses & Dissertations Project

Title:	Advances in deep learning with applications to computer vision and astronomy
Author:	Hu, Zhihang
View Online:	njit-etd2021-085 (xiv, 84 pages ~ 14.4 MB pdf)
Department:	Department of Computer Science
Degree:	Doctor of Philosophy
Program:	Computer Science
Document Type:	Dissertation
Advisory Committee:	Wang, Jason T. L. (Committee chair) Koutis, Ioannis (Committee member) Li, Jing (Committee member) Herbert, Katherine Grace (Committee member) Wang, Haimin (Committee member)
Date:	2021-08
Keywords:	Computer vision Deep learning Generative adversarial nets Generative models Image processing Video processing
Availability:	Unrestricted
Abstract:	Deep Learning has spanned a variety of applications in computer vision as well as computational astronomy. These two aspects obtained similar data structure, therefore, their solutions can be transferable between each other. This dissertation look into two video-related tasks in computer vision and propose a novel problem in computational astronomy. Specifically, acquiring an in-depth understanding of videos has been a cornerstone problem in computer vision. This problem has been studied by various researchers from different perspectives, among which video prediction has attracted much attention. Video prediction aims to generate the pixels of future frames given a sequence of context frames. In practice, unlabeled video sequences can be gathered autonomously from a sensor or recording device. A machine capable of predicting future events using these video sequences in an unsupervised manner will gain extensive and deep knowledge about its physical environment and surroundings. However, despite its appealing prospects, accurate video prediction remains an open problem. The major challenge is the inherent uncertainty in the dynamics of the world. A typical example is that the future trajectory of a ball hitting the ground is inherently random. This dissertation proposes new generative adversarial networks (GANs) for stochastic video prediction. The proposed framework, dubbed Video-Prediction-GAN(VPGAN), employs an adversarial inference model and a cycle-consistency loss function to empower the framework to obtain more accurate predictions. In addition, a conformal mapping network structure is incorporated into VPGAN to enable action control for generating desirable future frames. In this way, VPGAN can produce fake videos of an object moving along a specific direction. Experimental results based on different datasets demonstrate the good performance of VPGAN and its superiority over existing methods. Other contributions of this dissertation include the development of a Three-Dimensional atrous convolutional long short-term memory network for background subtraction used in video processing and an extension of VPGAN for synthesizing vector magnetograms of active regions in different solar cycles. The dissertation concludes by pointing out some directions of future research in applying deep learning to computer vision and astronomy.