Faster and Cheaper Training for Large Models

Faster and Cheaper Training for Large Models

Two broad lines of research to make large-scale ML accessible: (1) pipeline and hybrid parallelism as in PipeDream and FlexFlow for cheaper training of existing DNN models; (2) retrieval-based NLP models like ColBERT that search through a corpus of documents at inference time rather than memorizing all knowledge in parameters.