NLP and Society: A Perspective from sentiment and emotion analysis, and mental health monitoring

Monday, September 19, 2022 - 10:00 am
online

Bio: Dr. Pushpak Bhattacharyya is a Professor of Computer Science and Engineering at IIT Bombay. He has done extensive research in Natural Language Processing and Machine Learning. Some of his noteworthy contributions are IndoWordnet, Eye Tracking assisted NLP, Low Resource MT, Multimodal multitasked multilingual sentiment and emotion analysis, and Knowledge Graph-Deep Learning Synergy in Information Extraction and Question Answering. He has published close to 400 research papers, has authored/co-authored 6 books including a textbook on machine translation, and has guided more than 350 students for their Ph.D., master's, and Undergraduate thesis. Prof. Bhattacharyya is a Fellow of the National Academy of Engineering, Abdul Kalam National Fellow, Distinguished Alumnus of IIT Kharagpur, past Director of IIT Patna, and past President of ACL. http://www.cse.iitb.ac.in/~pb

Abstract: In this talk, we describe our long-standing work on sentiment and emotion analysis, and also the use of NLP for mental health monitoring. The last mentioned is a stigma that society prefers to keep under cover, but with hazardous consequences. We describe our contribution to the techniques of sentiment and emotion analysisoften multimodal, multitasking, and multilingual. Such techniques have also proven useful in monitoring mental distress and providing positivity and hope through NLP agents. The work reported is the result of efforts of generations of students, and has found a place in top journals and conferences.

Can we ever trust our chatbots? Towards trustable collaborative assistants

Friday, September 16, 2022 - 02:20 pm
Storey Innovation Center 1400

In-Person Meeting Location:

Storey Innovation Center 1400

Live Meeting Link for Virtual Audience:

Abstract: 

AI services are known to have unstable behavior when subjected to changes in data, models or users. Such behaviors, whether triggered by omission or commission, lead to trust issues when AI work with humans. The current approach of assessing AI services in a black box setting, where the consumer does not have access to the AI’s source code or training data, is limited. The consumer has to rely on the AI developer’s documentation and trust that the system has been build as stated. Further, if the AI consumer reuses the service to build other services which they sell to their customers, the consumer is at the risk of the service providers (both data and model providers).

In this talk, I will cover chatbots (collaborative assistants), the problem of trust in this context and how one may make them more trustable. We will cover software testing, AI robustness, randomized control trial and the idea of rating AI based on their behavior. I will highlight some of our work, present key results and discuss ongoing work.

Speaker's Bio:

Biplav Srivastava is a Professor of Computer Science at the AI Institute at the University of South Carolina. Previously, he was at IBM for nearly two decades in the roles of a Research Scientist, Distinguished Data Scientist and Master Inventor. Biplav is  an ACM Distinguished Scientist, AAAI Senior Member, IEEE Senior Member and AAAS Leshner Fellow for Public Engagement on AI (2020-2021). His focus is on promoting goal-oriented, ethical, human-machine collaboration via natural interfaces using  domain and user models, learning and planning. Biplav has been working in AI trust for the last 3 years pursuing ideas in AI testing,

rating, randomized control and adversarial learning. He applies these techniques in areas of social as well as commercial relevance with particular attention to issues of developing countries  (e.g., transportation, water, health and governance). Biplav’s work has lead to many science firsts and high-impact commercial innovations ($B+), 190+ papers and 60+ US patents issued,

and awards for papers, demos and hacks. He has interacted with commercial customers, universities and governments, been on multilateral bodies, and assisted business leaders on technical issues.

More details about him are at: https://sites.google.com/site/biplavsrivastava/

Women in Computing: Interview Prep

Monday, September 12, 2022 - 06:00 pm
Room 2277 at the Story Innovation Center

Women in Computing will be hosting its first meeting of the Fall semester at 6pm next Monday, in Room 2277 at the Story Innovation Center! Women in Computing is open to all majors and everyone – all genders and majors is welcome! Please see the following message for details.

AI x Mathematics

Friday, September 9, 2022 - 02:20 pm
online

Virtual Meeting Link

Abstract: 

For the past few years, we have been intensively collaborating with mathematicians from Oxford and the University of Sydney. As a product of this collaboration, we have successfully demonstrated that analysing and interpreting the outputs of (graph) neural networks -- trained on carefully chosen problems of interest -- can directly assist mathematicians with proving difficult theorems and conjecturing new approaches to long-standing open problems. Specifically, using our method we have independently derived novel top-tier mathematical results in areas as diverse as representation theory and knot theory.

By doing so, we present AI as the mathematician's "pocket calculator of the 21st century". The significance of our result has been recognised by the journal Nature, where our work featured on the cover page.

In this talk, I aim to tell you all about our findings, from a personal perspective. Expect key details of our modelling work + an account of how it felt to interact with top mathematicians.

Speaker's Bio: Please see https://petar-v.com/ for a short bio.

AI for science:  How machine learning and deep learning are transforming materials discovery

Friday, September 2, 2022 - 02:20 pm
Storey Innovation Center 1400

In-Person Meeting Location:

Storey Innovation Center 1400

Live Meeting Link for Virtual Audience

Talk Abstract: Artificial intelligence and deep learning are increasingly transforming all scientific disciplines with their superior capability to learn to detect patterns from large amount of data and to learn predictive models from data without relying upon theory or deep mechanistic understanding, with their capability to build generative models for inverse design of materials and molecules and with the models to generate synthetic data. In this talk, we present our research focusing on using deep learning and machine learning to discover and model the patterns in and relationships of structures and functions in materials and molecules and how to exploit such learned dark/implicit knowledge in deep learning based generative design of novel materials, graph neural network based materials property prediction, and deep learning based crystal structure prediction of inorganic materials. Considering that the number of inorganic materials discovered so far (~250,000) by humanity is only a tiny portion of the almost infinite chemical design space, our AI based data-driven computational materials discovery has the potential to transform the conventional trial-and-error approaches in materials discovery.

Speaker's Bio: Dr. Jianjun Hu is currently a Full Professor of computer science at the Department of Computer Science and Engineering, University of South Carolina, Columbia SC. He was associate professor from 2013 to 2022 and assistant professor from 2007 to 2013 at the same department. Dr. Hu received his B.S. and M.S. degrees of Mechanical Engineering in 1995 and 1998 respectively from Wuhan University of Technology, China. He received the Ph.D. degree of Computer Science in 2004 from Michigan State University in the area of machine learning and evolutionary computation, under the supervision of Professor Erik Goodman. He then worked as Postdoctoral Fellow at Purdue University with Prof. Daisuke Kihara and University of Southern California with Prof. Xianghong Zhou from 2004 to 2007 in the area of bioinformatics. Dr. Hu’s main research has focused on machine learning, deep learning, evolutionary computation and their applications in materials informatics, bioinformatics, engineering design, and intelligent manufacturing. His works have been published in PNAS, Advanced Science, Nature npj Computational Materials, Patterns (Cell Press), Evolutionary Computation Journal, Journal of physical chemistry, Scientific Report, and so on with a total of more than 200 journal and conference papers (H-index 35 with > 4200 citations). Currently, his main research is focused on utilizing deep learning to discover the relationship of structures and functions in materials, molecules, and proteins and exploit the learned implicit knowledge for generative design of transformative new materials, drugs, and proteins. His work and research lab info can be found online at: http://mleg.cse.sc.edu/publication

Explainable Artificial Intelligence and the Rubik’s Cube

Friday, August 26, 2022 - 02:20 pm
Storey Innovation Center 1400

This Friday (8/26), from 2:20 pm - 3:10 pm, at the Seminar in Advances in Computing, Dr. Forest Agostinelli from UofSC will give an in-person talk entitled “Explainable Artificial Intelligence and the Rubik’s Cube”.

In-Person Meeting Location:

Storey Innovation Center 1400

Live Meeting Link for Virtual Audience:

https://teams.microsoft.com/l/meetup-join/19%3ameeting_Nzc3MThiZTktZDY0Zi00MDQzLWE5YmMtMzhiYjlhZTBiMjE3%40thread.v2/0?context=%7b%22Tid%22%3a%224b2a4b19-d135-420e-8bb2-b1cd238998cc%22%2c%22Oid%22%3a%2218da07c8-8a10-4930-a982-b5863c90ddf4%22%7d 

Talk Abstract: The field of artificial intelligence (AI) has allowed computers to learn to synthesize chemical compounds, fold proteins, and write code. However, these AI algorithms cannot explain the thought processes behind their decisions. Not only is explainable AI important for us to be able to trust it with delicate tasks such as surgery and disaster relief, but it could also help us obtain new insights and discoveries. In this talk, I will present DeepCubeA, an AI algorithm that can solve the Rubik’s cube, and six other puzzles, without human guidance. Next, I will discuss how we are building on this work to create AI algorithms that can both solve puzzles and explain their solutions in a manner that we can understand. Finally, I will discuss how this work relates to problems in the natural sciences.

Speaker's Bio: Forest Agostinelli is an assistant professor at the University of South Carolina. He received his B.S. from the Ohio State University, his M.S. from the University of Michigan, and his Ph.D. from the University of California, Irvine under Professor Pierre Baldi. His group conducts research in the fields of deep learning, reinforcement learning, search, explainability, bioinformatics, and neuroscience. His homepage is located at https://cse.sc.edu/~foresta/.

Image-Based Crack Detection by Extracting Depth of The Crack Using Machine Learning

Monday, July 11, 2022 - 03:00 pm

THESIS DEFENSE

Author : Nishat Tabassum

Advisor : Dr. Casey Cole

Date : July 11, 2022

Time 3:00 pm

Place : Virtual (Teams link below)

 

Meeting Link:here

 

Abstract

 Concrete structures have been a major aspect of social infrastructure since the 1700s, so it has been used for centuries. Concrete is used for the durability and support it provides to buildings and bridges. Assessing the state of these structures is important in preserving the longevity of structures and the safety of the public. Detecting cracks in their early stage allows repairs to be made without the need to replace the whole structure, so it reduces the cost. Traditional methods are slowly falling behind as technology advances and an increase in demand for a practical method of crack detection. This study aims to review the practicality of CNN for evaluating damages from cracks autonomously. In addition, many previous methods of crack detection such as traditional manual techniques, image processing techniques, and machine learning methods are discussed. These methods will be investigated to assess the results and effectiveness of each method. Four primary cracks and sixteen secondary cracks of varying depths were chosen to train the CNN model for binary classification of whether a crack is present. A database of images of concrete without cracks was utilized to train the CNN model to recognize the features of images with and without cracks. Multiclass CNN was trained with a dataset of known depths of cracks to predict the severity of damages and cracks. Few studies have been done on depth prediction of cracks, so the aim of this study is to suggest XGBoost of a regression model as an effective method of in-depth prediction. vi Test results show that both the CNN models produced high accuracy in crack identification and damage zone classification. So, it is an effective method that can be used by civil engineers to monitor the well-being of the concrete to reduce labor and increase time efficiency. In addition, the XGBoost of a regression model produced exemplary accuracy in results for predicting the depths of cracks. This demonstrates the possibility of crack depth prediction. Predicting the depths of cracks is important in gaining insight into the health of the structure and can help determine the severity of the cracks and damage to the structure.

On Incorporating the Stochasticity of Quantum Machine Learning into Classical Models   

Wednesday, July 6, 2022 - 03:00 pm

THESIS DEFENSE

Author : Joseph Lindsay

Advisor : Dr. Ramtin Zand

Date : July 6, 2022

Time 3:00 pm

Place : Virtual (Teams link below)

 

Meeting Link: click here

 

Abstract

While many of the most exciting quantum computing algorithms are currently impossible to be implemented until fault-tolerant quantum error correction is achieved, noisy intermediate-scale quantum (NISQ) devices allow for smaller scale applications that leverage the paradigm for speed-ups to be researched and realized. A currently popular application for these devices is quantum machine learning (QML). Recent works over the past few years indicate that QML algorithms can function just as well as their classical counterparts, and even outperform them in some cases. Many current QML models take advantage of variational quantum algorithm (VQA) circuits, given that their scale is typically small enough to be compatible with NISQ devices and the method of automatic differentiation for optimizing circuit parameters is familiar to machine learning. As with many skeptics on its benefits of quantum computing in general, there is some concern as to whether machine learning is the "best" use case for the advantages that NISQ devices make possible. To this end, the nature of this work is to investigate the utilization of stochastic methods inspired by QML in attempt to approach the reported successes in performance. Using the long short-term memory (LSTM) model as a case study and by analyzing the performance of classical, stochastic, and QML methods, this work aims to elucidate if it is possible to achieve QML's benefits on classical machines by incorporating aspects of its stochasticity.

Cross Domain Semantic Segmentation

Tuesday, June 7, 2022 - 09:00 am

DISSERTATION DEFENSE

Author : Xinyi Wu

Advisor : Dr. Song Wang

Date : June 7, 2022

Time 9:00 am

Place : Virtual (Zoom link below)

 

Meeting Link: https://zoom.us/j/98175586174?pwd=bEs2a1g0azBmUlhCNWluT2R2bDhidz09

 

Abstract

As a long-standing computer vision task, semantic segmentation is still extensively researched till now because of its importance to visual understanding and analysis. The goal of semantic segmentation is to classify each pixel of images based on the pre-defined classes. In the era of deep learning, convolutional neural networks largely improve the accuracy and efficiency of semantic segmentation. However, this success is achieved with two limitations: 1) a large-scale labeled dataset is required for training while the labeling process for this task is quite labor-intensive and tedious; 2) the trained deep networks can get promising results when testing on the same domain (i.e., intra-domain test) but might suffer from a large performance drop when testing on different domains (i.e.,  cross-domain test). Therefore, developing algorithms that can transfer knowledge from labeled source domains to unlabeled target domains is highly desirable to address these two limitations. 

In this research, we explore three settings of cross domain semantic segmentation conditioned on the use of different training data in the target domain: 1) the use of a sole unlabeled target image, 2) the use of multiple unlabeled target images, and 3) the use of unlabeled target videos, respectively. 

At the first part, we tackle the problem of one-shot unsupervised domain adaptation (OSUDA) for semantic segmentation where the segmentors only use one unlabeled target image during training. In this case, traditional unsupervised domain adaptation models usually fail since they cannot adapt to the target domain with over-fitting to one (or few) unlabeled target samples. To address this problem, existing OSUDA methods usually integrate a style-transfer module to perform domain randomization based on the unlabeled target sample, with which multiple domains around the target sample can be explored during training. However, such a style-transfer module relies on an additional set of images as style reference for pre-training and also increases the memory demand for domain adaptation. Here we propose a new OSUDA method that can effectively relieve such computational burden by making full use of the sole target image in two aspects: (1) implicitly stylizing the source domain in both image and feature levels; (2) softly selecting the source training pixels. Experimental results on two commonly-used synthetic-to-real scenarios demonstrate the effectiveness and efficiency of the proposed method.  

Secondly, we work on the problem of nighttime semantic segmentation which plays an equally important role as that of daytime images in autonomous driving but is much more challenging and less studied due to poor illuminations and arduous human annotations. Our proposed solution employs an adversarial training with a labeled daytime dataset and an unlabeled dataset that contains coarsely aligned day-night image pairs. The unlabeled daytime images from the target dataset serve as an intermediate domain to mitigate the difficulty in day-to-night adaption since they share similarities with the source in illumination pattern and contain the same static-category objects as the their nighttime counterparts. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method achieves state-of-the-art performance for nighttime semantic segmentation. 

Finally, we propose a domain adaptation method for video semantic segmentation, i.e.,  the target is in video format. Before our work, other works were achieving this goal by transferring the knowledge from the source domain of self-labeled simulated videos to the target domain of unlabeled real-world videos. In our work, we argue that it is not necessary to use a labeled video dataset as the source since the temporal continuity of video segmentation in the target domain can be estimated and enforced without reference to videos in the source domain. This motivates a new framework of Image-to-Video Domain Adaptive Semantic Segmentation (I2VDA), where the source domain is a set of images without temporal information. Under this setting, we bridge the domain gap via adversarial training based on only the spatial knowledge, and develop a novel temporal augmentation strategy, through which the temporal consistency in the target domain is well-exploited and learned. In addition, we introduce a new training scheme by leveraging a proxy network to produce pseudo-labels on-the-fly, which is very effective to improve the stability of adversarial training. Experimental results on two synthetic-to-real scenarios show that the proposed I2VDA method can achieve even better performance on video semantic segmentation than existing state-of-the-art video-to-video domain adaption approaches. 

Image Restoration Under Adverse Illumination for Various Applications 

Monday, May 30, 2022 - 09:00 am

DISSERTATION DEFENSE

Author : Lan Fu

Advisor : Dr. Song Wang

Date : May 30, 2022

Time 9:00 am

Place : Virtual (Zoom link below)

Zoom link is : https://us05web.zoom.us/j/9860655563?pwd=Qld4ZUozUkFBSGFoa3lRZjNBN3ZVUT09 

 

Abstract

 

Many images are captured in sub-optimal environment, bringing about various kinds of degradations, such as noise, blur, and shadow. Adverse illumination is one of the most important factors resulting in image degradation with color and illumination distortion or even unidentified image content. Degradation caused by the adverse illumination makes the images suffer from worse visual quality, which might also lead to negative effects on high-level perception tasks, e.g., object detection.

 

Image restoration under adverse illumination is an effective way to remove such kind of degradations to obtain visual pleasing images. Existing state-of-the-art deep neural networks (DNNs) based image restoration methods have achieved impressive  performance for image visual quality improvement. However, different real-world applications require the image restoration under adverse illumination to achieve different goals. For example, in the computational photography field, visually pleasing image is desired in the smartphone photography. Nevertheless, for traffic surveillance and autonomous driving in the low light or nighttime scenario, high-level perception tasks, e.g., object detection, become more important to ensure safe and robust driving performance. Therefore, in this dissertation, we try to explore DNN-based image restoration solutions for images captured under adverse illumination in various applications: 1) image visual quality enhancement, 2) object detection improvement, and 3) enhanced image visual quality and better detection performance simultaneously.

 

First, in the computational photography field, a visually pleasing image is desired. We take shadow removal task as an example to fully explore image visual quality enhancement. Shadow removal is still a challenging task due to its inherent background-dependent and spatial-variant properties, leading to unknown and diverse shadow patterns. We propose a novel solution by formulating this task as an exposure fusion problem to address the challenges. We propose shadow-aware FusionNet to `smartly' fuse multiple over-exposure images with pixel-wise fusion weight maps, and boundary-aware RefineNet to eliminate the remaining shadow trace further. Experiment results show that our method outperforms other CNN-based methods in three datasets.

 

Second, we explore the application of CNN-based night-to-day image translation for vehicle detection improvement in the traffic surveillance field for safe and robust driving performance. We propose a detail-preserving method to implement the nighttime to daytime image translation and thus adapt daytime trained detection model to nighttime vehicle detection. We firstly utilize StyleMix method to acquire paired images of daytime and nighttime for following nighttime to daytime image translation training. The translation is implemented based on kernel prediction network to avoid texture corruption. Experimental results showed that the proposed method fit the nighttime vehicle detection task to reuse the daytime domain knowledge.

 

Third, we explore the image visual quality and facial landmark detection improvement simultaneously. For the portrait images captured in the wild, the facial landmark detection can be affected by the foreign shadow. We construct a novel benchmark SHAREL covering diverse face shadow patterns with different intensities, sizes, shapes, and locations to study the effects of shadow removal on facial landmark detection. Moreover, we propose a novel adversarial shadow attack to mine hard shadow patterns. We conduct extensive analysis on three shadow removal methods and three landmark detectors. Then, we design a novel landmark detection-aware shadow removal framework, which empowers shadow removal to achieve higher restoration quality and enhance the shadow robustness of deployed facial landmark detectors.