python多进程实战[入门篇]

    xiaoxiao2023-10-12  157

    一. 基于python内置库multiprocess.Process的多进程计算1. 实战:充分利用CPU的多核心、多进程计算1~10000000里的素数1.1 多进程 [os.cpu_count()=12] 计算的总耗时 [单位: 秒] 2. 实战:单进程计算1~10000000里的素数2.1 单进程计算的总耗时 [单位: 秒] 3. 结果分析:多进程和单进程的耗时对比:4. 本章测试代码所用的硬件、软件环境    

       前言:随着半导体工艺技术的飞速进步和体系结构的不断发展,CPU设计趋势已逐步从单核向多核演进,现今,多核处理器广泛普及,但实际上多核利用率不高,原因大多数是软件没有针对多核心优化。    本章实战多进程素数计算 ,对比单进程计算方式有着非常明显的效率提升,希望能给予你一定的启发。

    更多精彩,点我传送 >>>>> 梦马韶华主页    

    一. 基于python内置库multiprocess.Process的多进程计算

     

    1. 实战:充分利用CPU的多核心、多进程计算1~10000000里的素数

     

    # coding = utf-8 # @Author : TerryJay # @File : multi_process.py # @Software: PyCharm # @Time : 2019/5/24 21:54 # I believe that the God rewards the diligent. import os import time from multiprocessing import Process from math import sqrt class MyProcess(Process): def __init__(self, name, start_n, end_n): super().__init__() self.name = name self.start_n = start_n self.end_n = end_n self.calc_result = [] def loops_calc(self): for p in range(int(self.start_n), int(self.end_n)): if PrimeCalculation.is_prime(p): self.calc_result.append(p) return self.calc_result def run(self): print(self.name, "is run, start time is ---", time.time()) self.loops_calc() print(self.name, "is end, completion time is ---", time.time()) class PrimeCalculation: def __init__(self, number): self.process_num = os.cpu_count() self.number = number self.calc_block = self.number / self.process_num """任务切片,根据你的cpu实际线程数决定分成多少份,例如:4核8线程,则分为8份""" def task_assign(self): work_list = list() start_num = 1 slice_num = self.calc_block for p in range(self.process_num): work_list.append([start_num + p * slice_num, slice_num + p * slice_num]) return work_list """计算素数""" @staticmethod def is_prime(n): if n == 1: return False for num in range(2, int(sqrt(n)) + 1): if n % num == 0: return False return True if __name__ == '__main__': print(os.cpu_count()) main_start_time = time.time() f = PrimeCalculation(10000000) process_list = [] work_l = f.task_assign() for i in range(len(work_l)): m = MyProcess(str(i), work_l[i][0], work_l[i][1]) process_list.append(m) for prs in process_list: # prs.daemon = True prs.start() """主进程循环检查子进程活动状态,全部为False后程序退出""" while process_list: for prs in process_list: if not prs.is_alive(): process_list.remove(prs) print("所有进程已结束,程序总耗时>>>>>", time.time() - main_start_time)

    1.1 多进程 [os.cpu_count()=12] 计算的总耗时 [单位: 秒]

    所有进程已结束,程序总耗时>>>>> 27.244723081588745

       

    2. 实战:单进程计算1~10000000里的素数

     

    # coding = utf-8 # @Author : TerryJay # @File : single_process.py # @Software: PyCharm # @Time : 2019/5/24 21:54 # I believe that the God rewards the diligent. import time from math import sqrt class PrimeCalculation: @staticmethod def is_prime(n): if n == 1: return False for i in range(2, int(sqrt(n)) + 1): if n % i == 0: return False return True """计算素数""" @staticmethod def loops_calc(start_n, end_n): calc_result = [] start_time = time.time() for p in range(start_n, end_n): if PrimeCalculation.is_prime(p): calc_result.append(p) # print(calc_result) print("单进程计算已完成,总耗时>>>>>", time.time() - start_time) if __name__ == '__main__': PrimeCalculation.loops_calc(1, 10000000)

    2.1 单进程计算的总耗时 [单位: 秒]

    单进程计算已完成,总耗时>>>>> 125.80246877670288

     

    3. 结果分析:多进程和单进程的耗时对比:

    计算素数单进程计算耗时(s)多进程计算耗时(s)1~10000000125.802468776702889.129996061325073

     

    4. 本章测试代码所用的硬件、软件环境

    系统CPUpython版本MacOS 10.14.3i7 8700t 6c12t3.6.8
    最新回复(0)