Papers.

A Multi-Site Accelerator-Rich Processing Fabric for Scalable Brain-Computer Interfacing

Hull is an accelerator-rich distributed implantable Brain-Computer Interface (BCI) that reads biological neurons at data rates that are 2-3 orders of magnitude higher than the prior state of art, while supporting many neuroscientific applications. Prior approaches have restricted brain interfacing to tens of megabits per second in order to meet two constraints necessary for effective operation and safe long-term implantation -- power dissipation under tens of milliwatts and response latencies in the tens of milliseconds. Hull also adheres to these constraints, but is able to interface with the brain at much higher data rates, thereby enabling, for the first time, BCI-driven research on and clinical treatment of brain-wide behaviors and diseases that require reading and stimulating many brain locations. Central to Hull's power efficiency is its realization as a distributed system of BCI nodes with accelerator-rich compute. Hull balances modular system layering with aggressive cross-layer hardware-software co-design to integrate compute, networking, and storage. The result is a lesson in designing networked distributed systems with hardware accelerators from the ground up.

FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Models

Sequence-based deep learning recommendation models (DLRMs) are an emerging class of DLRMs showing great improvements over their prior sum-pooling based counterparts at capturing users' long term interests. These improvements come at immense system cost however, with sequence-based DLRMs requiring substantial amounts of data to be dynamically materialized and communicated by each accelerator during a single iteration. To address this rapidly growing bottleneck, we present FlexShard, a new tiered sequence embedding table sharding algorithm which operates at a per-row granularity by exploiting the insight that not every row is equal. Through precise replication of embedding rows based on their underlying probability distribution, along with the introduction of a new sharding strategy adapted to the heterogeneous, skewed performance of real-world cluster network topologies, FlexShard is able to significantly reduce communication demand while using no additional memory compared to the prior state-of-the-art. When evaluated on production-scale sequence DLRMs, FlexShard was able to reduce overall global all-to-all communication traffic by over 85%, resulting in end-to-end training communication latency improvements of almost 6x over the prior state-of-the-art approach.

GCS: Generalized Cache Coherence For Efficient Synchronization

We explore the design of scalable synchronization primitives for disaggregated shared memory. Porting existing synchronization primitives to disaggregated shared memory results in poor scalability with the number of application threads because they layer synchronization primitives atop cache-coherence substrates, which engenders redundant inter-core communications. Substantially higher cache-coherence latency ($\mu$s) with substantially lower bandwidths in state-of-the-art disaggregated shared memory designs amplifies the impact of such redundant communications and precludes scalability. In this work, we argue for a co-design for the cache-coherence and synchronization layers for better performance scaling of multi-threaded applications on disaggregated memory. This is driven by our observation that synchronization primitives are essentially a generalization of cache-coherence protocols in time and space. We present GCS as an implementation of this co-design. GCS employs wait queues and arbitrarily-sized cache lines directly at the cache-coherence protocol layer for temporal and spatial generalization. We evaluate GCS against the layered approach for synchronization primitives: the pthread implementation of reader-writer lock, and show that GCS improves in-memory key-value store performance at scale by 1 - 2 orders of magnitude.

Scaling Up Models and Data with t5x and seqio

Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures. $\texttt{t5x}$ and $\texttt{seqio}$ are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.

WORKING with Grep, Sed, and Awk Pocket Primer

What learning algorithm is in-context learning? Investigations with linear models.

What is the Inductive Bias of Flatness Regularization? A Study of Deep Matrix Factorization Models.

WandBot: GPT-4 Powered Chat Support

Toward L_∞Recovery of Nonlinear Functions: A Polynomial Sample Complexity Bound for Gaussian Random Fields.

Statistics Using Python

SkyPilot: An Intercloud Broker for Sky Computing

SHEPHERD: Serving DNNs in the Wild

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization.

Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models.

Robustness in Multimodal Learning under Train-Test Modality Mismatch

R for Deep Learning Pocket Primer

PyTorch RPC: Distributed Deep Learning Built on Tensor-Optimized Remote Procedure Calls

Distributed training technologies have advanced rapidly in the past few years and have unlocked unprecedented scalability with increasingly complex solutions. These technologies have made distributed training much more efficient and accessible, though they impose specific constraints on the training paradigm or the model structure.

Python for Absolute Beginners

Python 3 Data Visualization Using ChatGPT / GPT-4

Python 3 and Feature Engineering

Python 3 and Data Visualization

Prompt Engineering LLMs with LangChain and W&B

Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence.

Managing Datasets and Models

Making Data Engineering Declarative

Linux Shell Command Pocket Primer

JAVA Basics Using ChatGPT/GPT-4

Introducing LlamaIndex.TS

Intermediate Python

How Sharpness-Aware Minimization Minimizes Sharpness?

Harnessing Scala 3 metaprogramming without macros

Global Capacity Management With Flux

Functional Programming in Scala, Second Edition

First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains.

FARGO: Fast Maximum Inner Product Search via Global Multi-Probing

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining.

Data Structures in Java

Data Selection for Language Models via Importance Resampling.

Creating, Storing, and Sharing Your Own Web Archives with Open Source Webrecorder Tools

Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback

ChatGPT’s Knowledge is Two Years Old: What to do if you’re building LLM applications

Build production-ready generative AI applications for enterprise search using Haystack pipelines and Amazon SageMaker JumpStart with LLMs

Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time.

AutoKeras: An AutoML Library for Deep Learning

Asymptotic Instance-Optimal Algorithms for Interactive Decision Making.

Angular 13/14 Pocket Primer

A theoretical study of inductive biases in contrastive learning.

Autonomous Vehicles: Open-Source Technologies, Considerations, and Development

Autonomous vehicles are the culmination of advances in many areas such as sensor technologies, artificial intelligence (AI), networking, and more. This paper will introduce the reader to the technologies that build autonomous vehicles. It will focus on open-source tools and libraries for autonomous vehicle development, making it cheaper and easier for developers and researchers to participate in the field. The topics covered are as follows. First, we will discuss the sensors used in autonomous vehicles and summarize their performance in different environments, costs, and unique features. Then we will cover Simultaneous Localization and Mapping (SLAM) and algorithms for each modality. Third, we will review popular open-source driving simulators, a cost-effective way to train machine learning models and test vehicle software performance. We will then highlight embedded operating systems and the security and development considerations when choosing one. After that, we will discuss Vehicle-to-Vehicle (V2V) and Internet-of-Vehicle (IoV) communication, which are areas that fuse networking technologies with autonomous vehicles.

Implementation Strategies for Mutable Value Semantics.

Mutable value semantics is a programming discipline that upholds the independence of values to support local reasoning. In the discipline's strictest form, references become second-class citizens: they are only created implicitly, at function boundaries, and cannot be stored in variables or object fields. Hence, variables can never share mutable state. Unlike pure functional programming, however, mutable value semantics allows part-wise in-place mutation, thereby eliminating the memory traffic usually associated with functional updates of immutable data. This paper presents implementation strategies for compiling programs with mutable value semantics into efficient native code.

BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing

Recent work has shown that generation from a prompted or fine-tuned language model can perform well at semantic parsing when the output is constrained to be a valid semantic representation. We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output representations, as well as a constrained decoding interface to generate only valid outputs covered by these grammars. We provide low, medium, and high resource splits for each dataset, allowing accurate comparison of various language models under different data regimes. Our benchmark supports evaluation of language models using prompt-based learning as well as fine-tuning. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or surpass state-of-the-art methods for syntactic and semantic parsing when the model output is constrained to be valid.