java - Parallelization of map reduce -
i have 2 independent mapreduce jobs in single java program. how can make them run in parallel. system configuration hadoop 2.7.1 3 node 16 physical cores
instead of using job.waitforcompletion(true)
use job.submit()
method submitting job. later submit job , return immediately. job.waitforcompletion(true)
submit job , waits completion.
then should use iscomplete()
checking completion status of jobs. remember non-blocking call
, might want put while loop while sleep time.
job job1 = new job("job1"); job job2 = new job("job2"); .... // code configure both job objects. //now submit both jobs. job1.submit(); job2.submit(); // wait completion. while(!job1.iscomplete() && !job2.iscomplete()) { thread.sleep(10000); }
Comments
Post a Comment