Retrieval Augmented Generation for Knowledge Intensive NLP Tasks

RAG is a pretrained model with a differentiable access mechanism to explicit non-parametric memory for natural language generation. The parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. There are two RAG formulations: one which conditions on the same retrieved passages across the whole generated sequence (RAG-sequence), and another which can use different passages per token (RAG-token). Better than standard NLG models like BART and T5 on tasks like Open-domain Question Answering (Natural Questions, TriviaQA, WebQuestions, CuratedTrec), Abstractive Question Answering (MSMARCO), Jeopardy Question Generation (SearchQA), Fact Verification (FEVER). In this video, I will talk about architecture of RAG model. We will also discuss how RAG compares with T5 and BART on question generation and question answering. For more details, please look at arxiv.org/pdf/2005.11401.pdf and ai.facebook.com/blog/retrieval-augmented-generatio… Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.

• #science_technology
• #deep_learning
• #deep_learning_for_nlp
• #natural_language_processing
• #natural_language_generation
• #retrieval_augmented_generation
• #mips
• #knowledge_retriever