HPML 2018

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Raul Puri, Robert Kirby, Nikolai Yakovenko, Bryan Catanzaro

Nvidia USA

Authors' preprint on arXiv

slides

Recent work has shown how to train Convolutional Neural Networks (CNNs) rapidly on large image datasets, then transfer the knowledge gained from these models to a variety of tasks. Following Radford et al, in this work, we demonstrate similar scalability and transfer for Recurrent Neural Networks (RNNs) for Natural Language tasks. By utilizing mixed precision arithmetic and a 32k batch size distributed across 128 NVIDIA Tesla V100 GPUs, we are able to train a character-level 4096-dimension multiplicative LSTM (mLSTM) for unsupervised text reconstruction over 3 epochs of the 40 GB Amazon Reviews dataset in four hours. This runtime compares favorably with previous work taking one month to train the same size and configuration for one epoch over the same dataset. Converging large batch RNN models can be challenging. Recent work has suggested scaling the learning rate as a function of batch size, but we find that simply scaling the learning rate as a function of batch size leads either to significantly worse convergence or immediate divergence for this problem. We provide a learning rate schedule that allows our model to converge with a 32k batch size. Since our model converges over the Amazon Reviews dataset in hours, and our compute requirement of 128 Tesla V100 GPUs, while substantial, is commercially available, this work opens up large scale unsupervised NLP training to most commercial applications and deep learning researchers. A model can be trained over most public or private text datasets overnight.

Accelerating deep neural network training for action recognition on a cluster of GPUs

Guojing Cong, Giacomo Domeniconi, Joshua Shapiro, Fan Zhou, Barry Chen

IBM TJ Watson, Georgia Tech, Lawrence Livermore National Laboratory

paper

slides

Due to the additional temporal dimension, large-scale video action recognition is even more challenging than image recognition and typically takes days to train on modern GPUs even for modest-sized datasets. We propose algorithms and techniques to accelerate training of deep neural networks for action recognition on a cluster of GPUs. In terms of convergence and scaling, our distributed training algorithm with adaptive batch size is provably superior to popular asynchronous stochastic gradient descent algorithms. The convergence analysis of our algorithm shows it is possible to reduce communication cost and at the same time minimize the number of iterations needed for convergence. We customize the Adam optimizer for our distributed algorithm to improve efficiency. In addition, we employ transfer-learning to further reduce training time while improving validation accuracy. Compared with the base-line single-GPU stochastic gradient descent implementation of the two-stream training approach, our implementation achieves super-linear speedups on 16 GPUs while improving validation accuracy. For the UCF101 and HMDB51 datasets, the validation accuracies achieved are 93.1% and 67.9% respectively. As far as we know, these are the highest accuracies achieved with the two-stream approach that does not involve computationally expensive 3D convolutions or pretraining on much larger datasets.

An argument in favor of strong scaling for deep neural networks with small datasets

Renato Cunha, Eduardo Rodrigues, Matheus Palhares Viana, Dario Augusto Borges Oliveira

IBM Research

Authors' preprint on arXiv

slides

In recent years, with the popularization of deep learning frameworks and large datasets, researchers have started parallelizing their models in order to train faster. This is crucially important, because they typically explore many hyperparameters in order to find the best ones for their applications. This process is time consuming and, consequently, speeding up training improves productivity. One approach to parallelize deep learning models followed by many researchers is based on weak scaling. The minibatches increase in size as new GPUs are added to the system. In addition, new learning rates schedules have been proposed to fix optimization issues that occur with large minibatch sizes. In this paper, however, we show that the recommendations provided by recent work do not apply to models that lack large datasets. In fact, we argument in favor of using strong scaling for achieving reliable performance in such cases. We evaluated our approach with up to 32 GPUs and show that weak scaling not only does not have the same accuracy as the sequential model, it also fails to converge most of time. Meanwhile, strong scaling has good scalability while having exactly the same accuracy of a sequential implementation.

Deep Learning on Large-scale Multicore Clusters

Kazumasa Sakiyama, Shinpei Kato, Yutaka Ishikawa, Atsushi Hori, Abraham Monrroy

The University of Tokyo, RIKEN, Nagoya University

paper

slides

Convolutional neural networks (CNNs) have achieved outstanding accuracy among conventional machine learning algorithms. Recent works have shown that large and complicated models which take significant cost for training are needed to get higher accuracy. To train these models efficiently in HPCs, many parallelization techniques for CNNs have been developed. However, most techniques are mainly targeting GPUs and parallelizations for CPUs are not fully investigated. This paper explores CNN training performance on large-scale multicore clusters by optimizing intra-node processing and applying techniques of inter-node parallelization for multiple GPUs. Detailed experiments conducted on state-of-the-art multi-core processors using the OpenMP API and MPI framework demonstrated that Caffe-based CNNs can be accelerated by using well-designed multithreaded programs. We achieved at most 1.64 times speedup in convolution operations with devised lowering strategy compared to conventional lowering and acquired 772 times speedup with 864 nodes compared to one node.

On the Resilience of RTL NN Accelerators: Fault Characterization and Mitigation

Behzad Salami, Osman Unsal, Adrian Cristal Kestelman

Barcelona Supercomputing Center

paper

slides

Machine Learning (ML) is making a strong resurgence in tune with the massive generation of unstructured data which in turn requires massive computational resources. Due to the inherently compute- and power-intensive structure of Neural Networks (NNs), hardware accelerators emerge as a promising solution. However, with technology node scaling below 10nm, hardware accelerators become more susceptible to faults, which in turn can impact the NN accuracy. In this paper, we study the resilience aspects of Register-Transfer Level (RTL) model of NN accelerators, in particular, fault characterization and mitigation. By following a High Level Synthesis (HLS) approach, first, we characterize the vulnerability of various components of RTL NN. We observed that the severity of faults depends on both i) application-level specifications, i.e., NN data (inputs, weights, or intermediate), NN layers, and NN activation functions, and ii) architectural-level specifications, i.e., data representation model and the parallelism degree of the underlying accelerator. Second, motivated by characterization results, we present a low-overhead fault mitigation technique that can efficiently correct bit flips, by 47.3% better than state-of-the-art methods.

t-SNE-CUDA: GPU-Accelerated t-SNE and its Applications to Modern Data

David Chan, Roshan Rao, Forrest Huang, John Canny

University of California, Berkeley

Authors' preprint on arXiv

slides

github code

Modern datasets and models are notoriously difficult to explore and analyze due to their inherent high dimensionality and massive numbers of samples. Existing visualization methods which employ dimensionality reduction to two or three dimensions are often inefficient and/or ineffective for these datasets. This paper introduces t-SNE-CUDA, a GPU-accelerated implementation of t-distributed Symmetric Neighbor Embedding (t-SNE) for visualizing datasets and models. t-SNE-CUDA significantly outperforms current implementations with 50-700x speedups on the CIFAR-10 and MNIST datasets. These speedups enable, for the first time, visualization of the neural network activations on the entire ImageNet dataset - a feat that was previously computationally intractable. We also demonstrate visualization performance in the NLP domain by visualizing the GloVe embedding vectors. From these visualizations, we can draw interesting conclusions about using the L2 metric in these embedding spaces. t-SNE-CUDA is publicly available at https://github.com/CannyLab/tsne-cuda.

HyperSpace: Distributed Bayesian Hyperparameter Optimization

M. Todd Young, Jacob Hinkle, Arvind Ramanathan, Ramakrishnan Kannan

Oak Ridge National Lab

paper

slides

github code

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.

A Machine Learning Approach for Parameter Screening in Earthquake Simulation

Marisol Monterrubio-Velasco, Jose Carlos Carrasco-Jiménez, Octavio Castillo-Reyes, Fernando Cucchietti, Josep De la Puente

Barcelona Supercomputing Center

paper

slides

Earthquakes are the result of rupture in the Earths crust. The rupture process is difficult to model deterministically due to the number of unmeasurable parameters involved and poorly constrained physical conditions, as well as the very diverse scales involved in their nucleation (meters) and complete failure (up to hundreds of kilometers). Hence computational physics has been only marginally successful at reproducing seismological observations so far. Our study makes use of synthetic seismic catalogs generated with a stochastic modelling technique called Fiber Bundle Model (FBM). Such catalogs can be readily compared with statistical values computed from real earthquake series, but the link between the FBM parametrization and the characteristics of the obtained earthquake series is difficult to assess. Furthermore, the stochastic nature of the model requires a large amount of realizations in order to attain sta- tistical robustness. In this study we show how high-performance computing (HPC) combined with Machine Learning are key enabling technologies to perform and improve the simulations, data management process and data analysis required to produce precise synthetic earthquake catalogs, analogous to the real ones. A total of three parameters are required by the Fiber Bundle Model: the grid size N, in which seismic events are simulated, the initial probability load P, and the percentage of dissipate load πf rac. We use a ML approach to study how the statistical patterns of synthetic data change according to these parameters and the geometrical fault configurations. We show that FBM grid size N is relevant when classifying synthetic catalogs generated using the FBM. Furthermore, in real seismic catalogs, the magnitude of completeness is crucial to determine an appropriate statistical behavior, therefore, we carry out an experiment to seek a proper equilibrium between N and the minimum magnitude generated. Last, we identify the most important statistical features that can be used to estimate the FBM optimal parameters with the aim of generating sequences that are similar to the real ones. We evaluate our results by comparing synthetic sequences to those obtained from three real earthquakes: Northridge, Landers, and Hector Mine.

A Case Study on Optimizing Accurate Half Precision Average

Kenny Peou, Joel Falcou, Alan Kelly

Numscale, Université Paris-Saclay

paper

slides

In this work, we study the numerical performance of various common algorithms used to calculate the average of an array of half precision (FP16) floating point values. While the current generation of CPUs does not support native FP16 arithmetic, it is a planned feature in a number of next-generation CPUs. FP16 arithmetic was emulated via the \texttt{half} software library. Due to the limitations of the FP16 data type, some algorithms proved insufficient for arrays as small as 100 elements. We propose an algorithm that allows numerically stable FP16 computation of the average and compare it to the naive floating point (FP32) algorithm in terms of both numerical precision and runtime performance. We find that our algorithm offers comparable numerical precision and SIMD performance to the higher precision computation.

Optimization of a sparse grid-based data mining kernel for architectures using AVX-512

Paul Cristian Sârbu, Hans-Joachim Bungartz

Technical University of Munich

paper

slides

Sparse grids have already been successfully used in various high-performance computing (HPC) applications, including data mining. In this article, we take a legacy classification kernel previously optimized for the AVX2 instruction set and investigate the benefits of using the newer AVX-512-based multi- and many-core architectures. In particular, the Knights Landing (KNL) processor is used to study the possible performance gains of the code. Not all kernels benefit equally from such architectures, therefore choices in optimization steps and KNL cluster and memory modes need to be filtered through the lens of the code implementation at hand. With a less traditional approach of manual vectorization through instruction-level intrinsics, our kernel provides a differently faceted look into the optimization process. Observations stem from results obtained for node- and cluster-level classification simulations with up to 2^28 multidimensional training data points, using the CooLMUC-3 cluster of the Leibniz Supercomputing Center (LRZ) in Garching, Germany.

Energy Efficient Parallel K-Means Clustering for an Intel® hybrid Multi-Chip Package

Matheus A. Souza, Lucas A. Maciel, Pedro H. Penna, Henrique C. Freitas

PUC Minas

paper

slides

FPGA devices have been proving to be good candidates to accelerate applications from different research topics. For instance, machine learning applications such as K-Means clustering usually relies on large amount of data to be processed, and, despite the performance offered by other architectures, FPGAs can offer better energy efficiency. With that in mind, Intel has launched a platform that integrates a multicore and an FPGA in the same package, enabling low latency and coherent fine-grained data offload. In this paper, we present a parallel implementation of the K-Means clustering algorithm, for this novel platform, using OpenCL language, and compared it against other platforms. We found that the CPU+FPGA platform was more energy efficient than the CPU-only approach from 70.71% to 85.92%, with Standard and Tiny input sizes respectively, and up to 68.21% of performance improvement was obtained with Tiny input size. Furthermore, it was up to 7.2x more energy efficient than an Intel Xeon Phi, 21.5x than a cluster of Raspberry Pi boards, and 3.8x than the low-power MPPA-256 architecture, when the Standard input size was used.

Performance Comparison of a Parallel Recommender Algorithm across three Hadoop-based Frameworks

Christina Diedhiou, Bryan Carpenter, Aamir Shafi, Soumabha Sarkar, Ramazan Esmeli, Ryan Gadsdon

University of Portsmouth, Imam Abdulrahman Bin Faisal University

paper

slides

One of the challenges our society faces is the ever-increasing amount of data, requiring systems to analyse large data sets without compromising their performances, and humans to navigate through a deluge of irrelevant material. Among existing platforms that address the system requirements, Hadoop is a framework widely used to store and analyse “big data”. On the human side, one of the aids to finding the things people really want is recommendation systems. This paper evaluates approaches to highly scalable parallel algorithms for recommendation systems with application to very large data sets. A particular goal is to evaluate an open source Java message passing library for parallel computing called MPJ Express, which has been integrated with Hadoop. As a demonstration we use MPJ Express to implement collaborative filtering on various data sets using the algorithm ALSWR (Alternating-Least-Squares with Weighted-λ-Regularization). We benchmark the performance and demonstrate parallel speedup on Movielens and Yahoo Music data sets. We then compare our results with two other frameworks: Mahout and Spark. Our results indicate that MPJ Express implementation of ALSWR has very competitive performance and scalability in comparison with the two other frameworks.

Effect Of Network Topology On The Performance Of ADMM-based SVMs

Shirin Tavara, Alexander Schliep

University of Boras, University of Gothenburg

paper

slides

Alternating Direction Method Of Multipliers (ADMM) is one of the promising frameworks for training Support Vector Machines (SVMs) on large-scale data in a distributed manner. In a consensus-based ADMM, nodes may only communicate with one-hop neighbors and this may cause slow convergence. In this paper, we investigate the impact of network topology on the convergence speed of ADMM-based SVMs using expander graphs. In particular, we investigate how much the expansion property of the network influence the convergence and which topology is preferable. Besides, we supply an implementation making these theoretical advances practically available. The results of the experiments show that graphs with large spectral gaps and higher degrees exhibit accelerated convergence.

High Performance Ensembles of Online Sequential Extreme Learning Machine for Regression and Time Series Forecasting

Luís F. L. Grim, André L. S. Gradvoh

University of Campinas

paper

slides

Ensembles of Online Sequential Extreme Learning Machine algorithm are suitable for forecasting Data Streams with Concept Drifts. Nevertheless, data streams forecasting require high-performance implementations due to the high incoming samples rate. In this work, we proposed to tune-up three ensembles, which operates with the Online Sequential Extreme Learning Machine, using high-performance techniques. We reimplemented them in the C programming language with Intel MKL and MPI libraries. Intel MKL provides functions that explore the multithread features in multicore CPUs, which expands the parallelism to multiprocessors architectures. MPI allows us to parallelize tasks with distributed memory on several processes, which can be allocated within a single computational node, or spread over several nodes. In summary, our proposal consists of a two-level parallelization, where we allocated each ensemble model into an MPI process, and we parallelized the internal functions of each model in a set of threads through Intel MKL. Thus, the objective of this work is to verify if our proposals provide a significant improvement in execution time when compared to the respective conventional serial approaches. For the experiments, we used a synthetic and a real dataset. Experimental results showed that, in general, the high-performance ensembles improve the execution time, when compared with its serial version, performing up to 10-fold faster.