3. Dask

Dask is a parallel computing framework for Python that can extend computational tasks to multi-core and cluster environments. This chapter will provide the reader with an understanding of the fundamental principles of Dask through several examples.

• 3.1. Dask Overview
• 3.2. Getting Started with Dask DataFrame
  • Creating Dask DataFrame
  • Trigger Computation
  • Index
  • Pandas Compatibility
  • Computational Graph
• 3.3. Scaling Dask to a Cluster
  • Dask Cluster
  • LocalCluster
  • Starting a Dask Cluster from the Command Line
  • Python Environment and Dependency Management
  • SSH, Kubernetes, High-Performance Computing Clusters
    • SSH
    • Kubernetes
    • High-Performance Computing (HPC) Clusters
  • Auto-Scaling
  • Dask Nanny
• 3.4. GPU
  • Dask GPU Cluster
  • GPU Task
    • Example: SVD
• 3.5. Task Graph and Data Partitioning
  • Task Graph
  • Data Partitioning
  • Data Partitioning Granularity
    • Data Blocks Too Small
    • Data Blocks Too Large
  • Iterative Algorithms
  • Setting the Right Data Block Size
    • Dask Dashboard
    • Setting the Data Block Size