ParallelPython 分布式计算模块
Parallel Python是Python进行分布式计算的开源模块,能够将计算压力分布到多核CPU或集群的多台计算机上,能够非常方便的在内网中搭建一个自组织的分布式计算平台。先从多核计算开始,普通的Python应用程序只能够使用一个CPU进程,而通过Parallel Python能够很方便的将计算扩展到多个CPU进程中
示例代码:
#!/usr/bin/python # File: sum_primes.py # Author: VItalii Vanovschi # Desc: This program demonstrates parallel computations with pp module # It calculates the sum of prime numbers below a given integer in parallel # Parallel Python Software: http://www.parallelpython.com import math, sys, time import pp def isprime(n): """Returns True if n is prime and False otherwise""" if not isinstance(n, int): raise TypeError("argument passed to is_prime is not of 'int' type") if n < 2: return False if n == 2: return True max = int(math.ceil(math.sqrt(n))) i = 2 while i <= max: if n % i == 0: return False i += 1 return True def sum_primes(n): """Calculates sum of all primes below given integer n""" return sum([x for x in xrange(2,n) if isprime(x)]) print """Usage: python sum_primes.py [ncpus] [ncpus] - the number of workers to run in parallel, if omitted it will be set to the number of processors in the system """ # tuple of all parallel python servers to connect with ppservers = () #ppservers = ("10.0.0.1",) if len(sys.argv) > 1: ncpus = int(sys.argv[1]) # Creates jobserver with ncpus workers job_server = pp.Server(ncpus, ppservers=ppservers) else: # Creates jobserver with automatically detected number of workers job_server = pp.Server(ppservers=ppservers) print "Starting pp with", job_server.get_ncpus(), "workers" # Submit a job of calulating sum_primes(100) for execution. # sum_primes - the function # (100,) - tuple with arguments for sum_primes # (isprime,) - tuple with functions on which function sum_primes depends # ("math",) - tuple with module names which must be imported before sum_primes execution # Execution starts as soon as one of the workers will become available job1 = job_server.submit(sum_primes, (100,), (isprime,), ("math",)) # Retrieves the result calculated by job1 # The value of job1() is the same as sum_primes(100) # If the job has not been finished yet, execution will wait here until result is available result = job1() print "Sum of primes below 100 is", result start_time = time.time() # The following submits 8 jobs and then retrieves the results inputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700) jobs = [(input, job_server.submit(sum_primes,(input,), (isprime,), ("math",))) for input in inputs] for input, job in jobs: print "Sum of primes below", input, "is", job() print "Time elapsed: ", time.time() - start_time, "s" job_server.print_stats() # Parallel Python Software: http://www.parallelpython.com
评论
libGlass分布式计算框架
libGlass提供了一组可伸缩的组件用来执行分布式计算。应用程序在需要的情况下被当作是可重用的组件。该框架适合新的应用程序,同时对一些老应用也同样可用,而无需去改写。
libGlass分布式计算框架
0
Tiny分布式计算框架
其于职业介绍所、工头、工人、工作模型的分布式计算框架。职业介绍所有两种,一种是本地职业介绍所,一种是远程职业介绍所。顾名思义,本地职业介绍所就是在当前计算机上的,远程职业介绍所用于连接到远程职业介绍所
Tiny分布式计算框架
0
Onyx分布式计算系统
Onyx是什么?Onyx是一个无中心、支持云、容错的分布式计算系统使用Clojure编写支持批处理和流处理混合提供信息模型用于描述和构建分布式工作流竞争对手:Storm,Cascading,Map/R
Onyx分布式计算系统
0