bash - Total Time computation of Map Reduce job in Hadoop-2.6 -


i want compute total(elapsed ) time of successful map reduce job( accurately) finished on hadoop -2.6. want pass time in shell script. know following ways compute finish time

  • by parsing web interface(http://192.168.1.169:8088/cluster/apps) html on job completion , extracting elapsed time.( problem in method times hadoop-2.6 web interface hangs long time. style not helpful me. not know why hangs ? )
  • i using bash script(as mentioned bellow) , writing time in file( finishtime). problem in additional delay of few second coming , time not giving correct results.

#!/bin/bash  e=$(date +%s%n | cut -b1-13)  a=($(mapred job -list |awk '{print $1," ",$2," ",$3}' | sort -k1 -n  | tail -n 3 | head -n 1))  wait    >finishtime  destdir=./finishtime  >destdir    if [ "${a[1]}" = "succeeded" ]           total=$(($e - ${a[2]}))          echo "$total" > "$destdir"  else          echo $(($(sed -n '1p' < ./init_setting) * 1000)) > "$destdir"  fi  exit

  • modification in job code , printing time on console( source code of job not available).

if know way of extracting elapsed time of job finished please notify me. if can extract via console.

do know hadoop command "history", can use command print details information of completed job.

the usage "bin/hadoop job -history ".“” on hdfs, job dir. e.g.

bin/hadoop job -history hdfs://172.16.19.5:9000/output/ >> test 

note replace path yours

the result on pictureenter image description here


Comments