Papers.

I'm So Glad I'm Uncoordinated: Coordination Is Increasingly Painful - What Can Be Done?

Keynote arguing that coordination is the dominant cost in scaled-out systems and surveying the ways to avoid it.

The OCEAN mailing list data set: Network analysis spanning mailing lists and code repositories

Communication surrounding the development of an open source project largely occurs outside the software repository itself. Historically, large communities often used a collection of mailing lists to discuss the different aspects of their projects. Multimodal tool use, with software development and communication happening on different channels, complicates the study of open source projects as a sociotechnical system. Here, we combine and standardize mailing lists of the Python community, resulting in 954,287 messages from 1995 to the present. We share all scraping and cleaning code to facilitate reproduction of this work, as well as smaller datasets for the Golang (122,721 messages), Angular (20,041 messages) and Node.js (12,514 messages) communities. To showcase the usefulness of these data, we focus on the CPython repository and merge the technical layer (which GitHub account works on what file and with whom) with the social layer (messages from unique email addresses) by identifying 33% of GitHub contributors in the mailing list data. We then explore correlations between the valence of social messaging and the structure of the collaboration network. We discuss how these data provide a laboratory to test theories from standard organizational science in large open source projects.

The Seattle report on database research

Cloud-Native Transactions and Analytics in SingleStore

Photon: A Fast Query Engine for Lakehouse Systems

Many organizations are shifting to a data management paradigm called the "Lakehouse," which implements the functionality of structured data warehouses on top of unstructured data lakes. This presents new challenges for query execution engines. The engine needs to provide good performance on the raw uncurated datasets that are ubiquitous in data lakes, and excellent performance on structured data stored in popular columnar file formats like Apache Parquet. Toward these goals, we present Photon, a vectorized query engine for Lakehouse environments that we developed at Databricks. Photon can outperform existing warehouses on SQL workloads and also supports the Apache Spark API. We discuss the design choices we made in Photon (e.g., vectorization vs. code generation) and describe its integration with our existing SQL and Apache Spark runtimes, its task model, and its memory manager. Photon has accelerated some customer workloads by over 10x and has recently allowed Databricks to set a new audited performance record for the official 100TB TPC-DS benchmark.

Finding Label and Model Errors in Perception Data With Learned Observation Assertions

TASTI: Semantic Indexes for Machine Learning-based Queries over Unstructured Data

Will This Online Shopping Session Succeed? Predicting Customer's Purchase Intention Using Embeddings

Bots don’t Vote, but They Surely Bother!

It takes a community---

A discussion with Reynold Xin, Wes McKinney, Alan Gates, and Chris McCubbin.

Jiffy: elastic far-memory for stateful serverless analytics

Stateful serverless analytics can be enabled using a remote memory system for inter-task communication, and for storing and exchanging intermediate data. However, existing systems allocate memory resources at job granularity---jobs specify their memory demands at the time of the submission; and, the system allocates memory equal to the job's demand for the entirety of its lifetime. This leads to resource underutilization and/or performance degradation when intermediate data sizes vary during job execution. This paper presents Jiffy, an elastic far-memory system for stateful serverless analytics that meets the instantaneous memory demand of a job at seconds timescales. Jiffy efficiently multiplexes memory capacity across concurrently running jobs, reducing the overheads of reads and writes to slower persistent storage, resulting in 1.6 -- 2.5× improvements in job execution time over production workloads. Jiffy implementation currently runs on Amazon EC2, enables a wide variety of distributed programming models including MapReduce, Dryad, StreamScope, and Piccolo, and natively supports a large class of analytics applications on AWS Lambda.

Ethical Challenges in AI

12th Temporal Web Analytics Workshop (TempWeb) Overview

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels. Though many IAA measures exist for simple categorical and ordinal labeling tasks, relatively little work has considered more complex labeling tasks, such as structured, multi-object, and free-text annotations. Krippendorff’s α, best known for use with simpler labeling tasks, does have a distance-based formulation with broader applicability, but little work has studied its efficacy and consistency across complex annotation tasks. We investigate the design and evaluation of IAA measures for complex annotation tasks, with evaluation spanning seven diverse tasks: image bounding boxes, image keypoints, text sequence tagging, ranked lists, free text translations, numeric vectors, and syntax trees. We identify the difficulty of interpretability and the complexity of choosing a distance function as key obstacles in applying Krippendorff’s α generally across these tasks. We propose two novel, more interpretable measures, showing they yield more consistent IAA measures across tasks and annotation distance functions.

Automatic Generation and Marking of UML Database Design Diagrams.

Textsetting as sequence alignment

Auditing saliency cropping algorithms

Training Data Subset Search With Ensemble Active Learning

Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could provide an effective solution to both improve performance and reduce training time. In this paper, we propose to scale up ensemble Active Learning (AL) methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data subset search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100 and ImageNet), as well as an internal object detection benchmark for prototyping perception models for autonomous driving. Unlike existing studies, our experiments on object detection are at the scale required for production-ready autonomous driving systems. We provide insights on the impact of different initialization schemes, acquisition functions and ensemble configurations at this scale. Our results provide strong empirical evidence that optimizing the training data distribution can provide significant benefits on large scale vision tasks.

Inferencing Big Data with Artificial Intelligence & Machine Learning Models in Metaverse

This quantitaive study provides different methods of visualization for processing Big Data sets in augmented and virtual reality. The goal is to provide the detailed implementation of the statistical methods and modeling techniques i.e. using machine learning algorithms and artificial intelligence. The Statistical analysis is performed on the Big Data sets in Metaverse, that points towards real time inferencing infrastructure and techniques. Also, This paper elaborates the importance of pervasive and Ubiquitous Computing Architecture and Applications that enable High-end computing on Big Data sets. In this paper, we evaluate the need for a combination of cognitive mechanism and high-end infrastructure to address the performance, latency, and security issues when working on Big Data sets in environments like virtual or Augmented reality. Despite the modernization and advancement in technology there are increasing number of breaches in cloud and hybrid infrastructures, this justifies the need to meet the requirements of strengthening security, safeguard mechanisms and privacy. This paper discusses methods beyond basic visualizations through a deep dive exploration on improvising the applications of existing analytical methods, and use of advanced exploratory tools and visualization techniques on the next generation platforms.

Evaluating operational readiness using chaos engineering simulations on Kubernetes architecture in Big Data

- How much confidence we can have in the interconnected complex systems that we put into production environment? In this paper we will provide the solution for operational readiness of a platform strengthening the backup, restore, network file transfer, failover capabilities and overall security. We provide the evaluation of inducing chaos to a Kubernetes environment which terminates random pods with data from edge devices in data centers while processing analytics on Big Data network and infer the recovery time of pods to calculate an estimated response time. In this research, we discuss the Operational Acceptance Testing through Chaos Engineering at all layers and more precisely on the modern architecture and practices like Microservices Architecture and Cloud computing which have changed our IT landscape in recent times.

Time-series ML-regression on Graphcore IPU-M2000 and Nvidia A100

A Comprehensive Evaluation of Novel AI Accelerators for Deep Learning Workloads

Reputation-Based Data Carrying for Web3 Networks

Distributed Attestation Revocation in Self-Sovereign Identity

Cough Diary based on Sound Classification, Source Validation and Event Detection

Coughing is a cardinal symptom of pulmonary and respiratory diseases, such as asthma, tuberculosis, chronic obstructive pulmonary disease, as well as coronavirus disease (COVID-19). The cough type, strength and frequency are indicators of the disease progression. Thus, several studies focused on the quantitative reporting of coughs through recording by a smart-phone and a sound classifier to provide a cough diary for a patient. However, those approaches report any cough, even coughs which are not caused by the patient. Thus, in this study, we aim to not only detect cough episodes, but also cough events and account coughs produced by the particular patient only. Accordingly, we report on an end-to-end solution for a patient cough diary consisting of three convolutional neural networks. The first recognizes respiratory sounds, including coughing by multi-class classification. The second validates if the cough was produced by the patient. It is based on a Siamese network using triplet-loss during training. Finally, individual cough events are detected by a cough onset classifier. For these three recognition models, we achieved an accuracy of 94%, 74%, and 94%, respectively. Furthermore, we explored the human-level performance of cough source validation through a field experiment involving 10 subjects. Our source validation model slightly outperformed the human cohort in the cough memorization task.

How to Boost Face Recognition with StyleGAN?

State-of-the-art face recognition systems require vast amounts of labeled training data. Given the priority of privacy in face recognition applications, the data is limited to celebrity web crawls, which have issues such as limited numbers of identities. On the other hand, self-supervised revolution in the industry motivates research on the adaptation of related techniques to facial recognition. One of the most popular practical tricks is to augment the dataset by the samples drawn from generative models while preserving the identity. We show that a simple approach based on fine-tuning pSp encoder for StyleGAN allows to improve upon the state-of-the-art facial recognition and performs better compared to training on synthetic face identities. We also collect large-scale unlabeled datasets with controllable ethnic constitution – AfricanFaceSet-5M (5 million images of different people) and AsianFaceSet-3M (3 million images of different people) – and we show that pretraining on each of them improves recognition of the respective ethnicities (as well as others), while combining all unlabeled datasets results in the biggest performance increase. Our self-supervised strategy is the most useful with limited amounts of labeled training data, which can be beneficial for more tailored face recognition tasks and when facing privacy concerns. Evaluation is based on a standard RFW dataset and a new large-scale RB-WebFace benchmark. The code and data are made publicly available at https://github.com/seva100/stylegan-for-facerec.

Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets.

Gromit: Benchmarking the Performance and Scalability of Blockchain Systems

SpaText: Spatio-Textual Representation for Controllable Image Generation

Recent text-to-image diffusion models are able to generate convincing results of unprecedented quality. However, it is nearly impossible to control the shapes of different regions/objects or their layout in a fine-grained fashion. Previous attempts to provide such controls were hindered by their reliance on a fixed set of labels. To this end, we present SpaText — a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map where each region of interest is annotated by a free-form natural language description. Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation, and show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-based. In addition, we show how to extend the classifier-free guidance method in diffusion models to the multi-conditional case and present an alternative accelerated inference algorithm. Finally, we offer several automatic evaluation metrics and use them, in addition to FID scores and a user study, to evaluate our method and show that it achieves state-of-the-art results on image generation with free-form textual scene control.

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework

Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks. In this paper, we present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint. The loss function is data driven and automatically adapts to arbitrary multi-label structures. Experiments on several datasets show that our relationship-preserving embedding performs well on a variety of tasks and outperform the base-line supervised and self-supervised approaches. Code is available at https://github.com/salesforce/hierarchicalContrastiveLearning.

Human vs. Artificial Intelligence

DeTrust-FL: Privacy-Preserving Federated Learning in Decentralized Trust Setting

Federated learning has emerged as a privacy-preserving machine learning approach where multiple parties can train a single model without sharing their raw training data. Federated learning typically requires the utilization of multi-party computation techniques to provide strong privacy guarantees by ensuring that an untrusted or curious aggregator cannot obtain isolated replies from parties involved in the training process, thereby preventing potential inference attacks. Until recently, it was thought that some of these secure aggregation techniques were sufficient to fully protect against inference attacks coming from a curious aggregator. However, recent research has demonstrated that a curious aggregator can successfully launch a disaggregation attack to learn information about model updates of a target party. This paper presents DeTrust-FL, an efficient privacy-preserving federated learning framework for addressing the lack of transparency that enables isolation attacks, such as disaggregation attacks, during secure aggregation by assuring that parties’ model updates are included in the aggregated model in a private and secure manner. DeTrust-FL proposes a decentralized trust consensus mechanism and incorporates a recently proposed decentralized functional encryption scheme in which all parties agree on a participation matrix before collaboratively generating decryption key fragments, thereby gaining control and trust over the secure aggregation process in a decentralized setting. Our experimental evaluation demonstrates that DeTrust-FL outperforms state-of-the-art FE-based secure multi-party aggregation solutions in terms of training time and reduces the volume of data transferred. In contrast to existing approaches, this is achieved without creating any trust dependency on external trusted entities.

TIFF: Tokenized Incentive for Federated Learning

In federated learning (FL), clients collectively train a global machine learning model with their own local data. Without sharing sensitive raw data, each client in FL only sends updated weights to consider privacy and security concerns. Most of existing FL works focus mainly on improving model accuracy and training time, but only a few works focus on FL incentive mechanisms. To build a high performance model after FL training, clients need to provide high quality and large amounts of data. However, in real FL scenarios, high-quality clients are reluctant to participate in FL process without reasonable compensation, because clients are self-interested and other clients can be business competitors. Even participation incurs some cost for contributing to the FL model with their local dataset. To address this problem, we propose TIFF, a novel tokenized incentive mechanism, where tokens are used as a means of paying for the services of providing participants and the training infrastructure. Without payment delays, participation can be monetized as both providers and consumers, which promotes continued long-term participation of high-quality data parties. Additionally, paid tokens are reimbursed to each client as consumers according to our newly proposed metrics (such as token reduction ratio and utility improvement ratio), which keeps clients engaged in FL process as consumers. To measure data quality, accuracy is calculated in training without additional overheads. We leverage historical accuracy records and random exploration to select high-utility participants and to prevent overfitting. Results show that TIFF provides more tokens to normal providers by up to 6.9% and less tokens to malicious providers by up to 18.1%, achieving improvement of the final model accuracy by up to 7.4%, compared to the default approach.

MeritRank: Sybil Tolerant Reputation for Merit-based Tokenomics

Exploration Trade-offs in Web Recommender Systems

Heterogeneity-Aware Adaptive Federated Learning Scheduling

Federated learning (FL) is becoming an important distributed machine learning approach that considers privacy and security concerns while training a shared model across various clients with localized data. One of the key challenges in FL is heterogeneity in both hardware resources and local datasets due to the nature of incorporating diverse clients. Given the resource heterogeneity, the availability of participating clients is not stable over time and their resource usage patterns become dynamic. This leads to resource wastage and straggler issues. Additional challenges are introduced due to data heterogeneity, causing model biasness and poor model performance. However, most existing FL systems are not well suited to heterogeneous environments because those approaches are not adaptive to various and dynamically changing resource usage patterns and accuracy trends during training process. To this end, we propose a heterogeneity-aware scheduling which is adaptive to the accuracy trends and various resource usage patterns. Our proposed scheduling provides different scheduling knobs for achieving different goals such as resource-efficient fast training, resource fairness, accuracy fairness, and high model performance. To the best of our knowledge, this is the first effort to mitigate effects of resource and data heterogeneity while providing adaptive scheduling based on dynamically changing resource usage patterns and accuracy trends.

Integrated Data Mapping Engine (DaME) for Financial Services

Enterprise organizations have vast datasets that need comprehensive analysis on a frequent basis, in order to manage data and take business decisions based on it. However, we observe that there can be a lack of industry standards for definitions of key terms. Additionally, there is a lack of governance for maintaining business processes. This typically leads to disconnected siloed datasets generated from disintegrated systems. To address these challenges, we developed a novel, integrated methodology DaME (Data Mapping Engine) that performs data mapping using ensemble of NLP techniques.The results from the industrial application and evaluation of DaME on a financial services dataset are encouraging that it can help reduce manual effort by automating data mapping and reusing the learning. The accuracy from our dataset in the application is much higher at 69% compared to the existing state-of-the-art with an accuracy of 34%. It has also helped improve the productivity of the industry practitioners, by saving them 14,000 hours of time spent manually mapping vast data stores over a period of ten months.

Association of upper respiratory Streptococcus pneumoniae colonization with SARS-CoV-2 infection among adults

Latent Factors of Language Disturbance and Relationships to Quantitative Speech Features

Association of Upper Respiratory Streptococcus pneumoniae Colonization With Severe Acute Respiratory Syndrome Coronavirus 2 Infection Among Adults

Towards a scalable discrete quantum generative adversarial neural network

Quantum generative adversarial networks (QGANs) have been studied in the context of quantum machine learning for several years, but there has not been yet a proposal for a fully QGAN with both, a quantum generator and discriminator. We introduce a fully QGAN intended for use with binary data. The architecture incorporates several features found in other classical and quantum machine learning models, which up to this point had not been used in conjunction. In particular, we incorporate classical noise reuploading in the generator, auxiliary qubits in the discriminator to enhance expressivity, and a direct connection between the generator and discriminator circuits, obviating the need to access the generator’s probability distribution. We show that, as separate components, the generator and discriminator perform as desired. We empirically demonstrate the expressive power of our model on both synthetic data as well as low energy states of an Ising model. Our demonstrations suggest that the model is not only capable of reproducing discrete training data, but also of potentially generalizing from it.

A 680,000-person megastudy of nudges to encourage vaccination in pharmacies

Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most-effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top-performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was "waiting for you." Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.

Explaining machine learning models with interactive natural language conversations using TalkToModel

Practitioners increasingly use machine learning (ML) models, yet models have become more complex and harder to understand. To understand complex models, researchers have proposed techniques to explain model predictions. However, practitioners struggle to use explainability methods because they do not know which explanation to choose and how to interpret the explanation. Here we address the challenge of using explainability methods by proposing TalkToModel: an interactive dialogue system that explains ML models through natural language conversations. TalkToModel consists of three components: an adaptive dialogue engine that interprets natural language and generates meaningful responses; an execution component that constructs the explanations used in the conversation; and a conversational interface. In real-world evaluations, 73% of healthcare workers agreed they would use TalkToModel over existing systems for understanding a disease prediction model, and 85% of ML professionals agreed TalkToModel was easier to use, demonstrating that TalkToModel is highly effective for model explainability. To ensure that a machine learning model has learned the intended features, it can be useful to have an explanation of why a specific output was given. Slack et al. have created a conversational environment, based on language models and feature importance, which can interactively explore explanations with questions asked in natural language.

Advances, challenges and opportunities in creating data for trustworthy AI

Rapid detection of SARS-CoV-2 RNA in saliva via Cas13

Who does what to whom? graph representations of action-predication in speech relate to psychopathological dimensions of psychosis

Accelerating material design with the generative toolkit for scientific discovery

With the growing availability of data within various scientific domains, generative models hold enormous potential to accelerate scientific discovery. They harness powerful representations learned from datasets to speed up the formulation of novel hypotheses with the potential to impact material discovery broadly. We present the Generative Toolkit for Scientific Discovery (GT4SD). This extensible open-source library enables scientists, developers, and researchers to train and use state-of-the-art generative models to accelerate scientific discovery focused on organic material design.

An End‐To‐End Earthquake Detection Method for Joint Phase Picking and Association Using Deep Learning

Foundations of Data Science

This chapter first defines data science, its primary objectives, and several related terms. It continues by describing the evolution of data science from the fields of statistics, operations research, and computing. The chapter concludes with historical notes on the emergence of data science and related topics.