Papers.

Do ImageNet Classifiers Generalize to ImageNet?

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

The State of Sparsity in Deep Neural Networks

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling.

Improving Semantic Parsing for Task Oriented Dialog

Semantic parsing using hierarchical representations has recently been proposed for task oriented dialog with promising results [Gupta et al 2018]. In this paper, we present three different improvements to the model: contextualized embeddings, ensembling, and pairwise re-ranking based on a language model. We taxonomize the errors possible for the hierarchical representation, such as wrong top intent, missing spans or split spans, and show that the three approaches correct different kinds of errors. The best model combines the three techniques and gives 6.4% better exact match accuracy than the state-of-the-art, with an error reduction of 33%, resulting in a new state-of-the-art result on the Task Oriented Parsing (TOP) dataset.

Cloud Programming Simplified: A Berkeley View on Serverless Computing

Serverless cloud computing handles virtually all the system administration operations needed to make it easier for programmers to use the cloud. It provides an interface that greatly simplifies cloud programming, and represents an evolution that parallels the transition from assembly language to high-level programming languages. This paper gives a quick history of cloud computing, including an accounting of the predictions of the 2009 Berkeley View of Cloud Computing paper, explains the motivation for serverless computing, describes applications that stretch the current limits of serverless, and then lists obstacles and research opportunities required for serverless computing to fulfill its full potential. Just as the 2009 paper identified challenges for the cloud and predicted they would be addressed and that cloud use would accelerate, we predict these issues are solvable and that serverless computing will grow to dominate the future of cloud computing.

Effectiveness of LSTMs in Predicting Congestive Heart Failure Onset

In this paper we present a Recurrent neural networks (RNN) based architecture that achieves an AUCROC of 0.9147 for predicting the onset of Congestive Heart Failure (CHF) 15 months in advance using a 12-month observation window on a large cohort of 216,394 patients. We believe this to be the largest study in CHF onset prediction with respect to the number of CHF case patients in the cohort and the test set (3,332 CHF patients) on which the AUC metrics are reported. We explore the extent to which LSTM (Long Short Term Memory) based model, a variant of RNNs, can accurately predict the onset of CHF when compared to known linear baselines like Logistic Regression, Random Forests and deep learning based models such as Multi-Layer Perceptron and Convolutional Neural Networks. We utilize demographics, medical diagnosis and procedure data from 21,405 CHF and 194,989 control patients to as our features. We describe our feature embedding strategy for medical diagnosis codes that accommodates the sparse, irregular, longitudinal, and high-dimensional characteristics of EHR data. We empirically show that LSTMs can capture the longitudinal aspects of EHR data better than the proposed baselines. As an attempt to interpret the model, we present a temporal data analysis-based technique on false positives to attribute feature importance. A model capable of predicting the onset of congestive heart failure months in the future with this level of accuracy and precision can support efforts of practitioners to implement risk factor reduction strategies and researchers to begin to systematically evaluate interventions to potentially delay or avert development of the disease with high mortality, morbidity and significant costs.

Equivariant Transformer Networks

Numerically Recovering the Critical Points of a Deep Linear Autoencoder

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Verified Uncertainty Calibration.

Universal Dependencies 2.5

Transparent Polynomial Commitment Scheme with Polylogarithmic Communication Complexity

Towards Taming the Resource and Data Heterogeneity in Federated Learning

Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks.

The I's Have It: Identity, Immutability, Idempotence, and Interchangeability Form the Backbone of Distributed Computing

Text Summarization of Product Titles.

TensorFlow Pocket Primer

TensorFlow 2 Pocket Primer

Stream Processing with Apache Spark

Stamping Out Counterfeit Goods with Blockchain and IoT

Simple Objectives Work Better

Rehashing Kernel Evaluation in High Dimensions

Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel.

RedShift: Transparent SNARKs from List Polynomial Commitment IOPs

Real-World Robotic Perception and Control Using Synthetic Data

Modern deep learning techniques are data-hungry, which presents a problem in robotics because real-world robotic data is difficult to collect. Simulated data is cheap and scalable, but jumping the "reality gap" to use simulated data for real-world tasks is challenging. In this thesis, we discuss using synthetic data to learn visual models that allow robots to perform manipulation tasks in the real world. We begin by discussing domain randomization, a technique for bridging the reality gap by massively randomizing the visual properties of the simulator. We demonstrate that, using domain randomization, synthetic data alone can be used to train a deep neural network to localize objects accurately enough for a robot to grasp them in the real world. The remainder of the thesis discusses extensions of this approach to a broader range of objects and scenes. First, we introduce a data generation pipeline inspired by the success of domain randomization for visual data that creates millions of unrealistic procedurally generated random objects, removing the assumption that 3D models of the objects are present at training time. Second, we reformulate the problem from pose prediction to grasp prediction and introduce a generative model architecture that learns a distribution over grasps, allowing our models to handle pose ambiguity and grasp a wide range of objects with a single neural network. Third, we introduce an attention mechanism for 3-dimensional data. We demonstrate that this attention mechanism can be used to perform higher fidelity neural rendering, and that models learned this way can be fine-tuned to perform accurate pose estimation when the camera intrinsics are unknown at training time.We conclude by surveying recent applications and extensions of domain randomization in the literature and suggesting several promising directions for research in sim-to-real transfer for robotics.

Python for TensorFlow Pocket Primer

Presto: SQL on Everything

Opportunities for Data Management Research in the Era of Horizontal AI/ML

On the Performance of Thompson Sampling on Logistic Bandits.

NIST TAC SM-KBP 2019 System Description: JHU/UR Framework

LIT: Learned Intermediate Representation Training for Model Compression

Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling.

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss.

Introduction to the special issue on annotated corpora

Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin.

Fixup Initialization: Residual Learning Without Normalization.

Equivariant Transformer Networks.

Dolphin : Dataplane Load-balancing in Programmable Hybrid Networks

DeepHashing using TripletLoss

Decoding AI in Financial Services: Business Implications for Boards and Professionals

Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation.

Connected Devices—Privacy and Cybersecurity Legal Issues

Confluo: Distributed Monitoring and Diagnosis Stack for High-speed Networks

Commonsense and Semantic-Guided Navigation through Language in Embodied Environment.

Challenges and Opportunities in DNN-Based Video Analytics: A Demonstration of the BlazeIt Video Query Engine.

C Programming Pocket Primer

Bootstrapping the Expressivity with Model-based Planning.

BIP 53: Disallow 64-byte transactions

Autolykos: The Ergo Platform PoW Puzzle

Approximability of Discriminators Implies Diversity in GANs.