1.2. Serial Execution v.s. Parallel Execution

If computing tasks are not parallelized, most of them are executed serially, as shown in Fig. 1.2. Here, the worker can be a computing core or a node in a cluster.

Fig. 1.2 Timeline Illustrating Serial Execution

In serial execution, tasks are executed one after another, with each task depending on the completion of the previous task. This sequential execution can lead to longer overall execution time, as tasks cannot be executed concurrently.

Cluster and heterogeneous computing provide more available computing cores, and parallel programming distributes computing tasks to multiple workers, as shown in Fig. 1.3. Whether it is multi-core on a single machine or multi-machine cluster, a scheduler is needed to distribute computing tasks to different workers. With more workers involved, the total time is reduced, and the saved time can be used for other tasks.

Fig. 1.3 Timeline Illustrating Distributed Execution