[5주차] Representation Learning with Contrastive Predictive Coding

[wavy-jung]

Keywords

CPC, InfoNCE, Predictive Coding, Negative Sampling

TL;DR

• Predictive coding과 NCE를 활용하여 여러 도메인에 적용될 수 있는 unsupervised 학습 방법 제시
• 기존 binary 비교 방식의 NCE에서 확장하여 한 입력 데이터 내에서 복수의 negative sample들과 하나의 positive sample을 비교하는 방식의 InfoNCE 제시
• 정보 이론 관점에서 negative sampling에 대한 수학적 분석

Abstract

While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Paper link

https://arxiv.org/abs/1807.03748

Presentation link

https://docs.google.com/presentation/d/1QDXmJL5YvycXf8vL-OYP61FKVTUecNqk/edit?usp=sharing&ouid=114847754426815005538&rtpof=true&sd=true

video link

https://youtu.be/vgzDpgxDVGQ

[jwkanggist]

@Doohae 님 감사합니다

혹시 13페이지 이후 수정이 안됐을까요 ?

[wavy-jung]

@Doohae 님 감사합니다

혹시 13페이지 이후 수정이 안됐을까요 ?

제가 수정 전 링크를 올렸었네요🥲 링크 수정해서 업데이트했습니다!