i want compute total(elapsed ) time of successful map reduce job( accurately) finished on hadoop -2.6. want pass time in shell script. know following ways compute finish time
- by parsing web interface(
http://192.168.1.169:8088/cluster/apps
) html on job completion , extracting elapsed time.( problem in method times hadoop-2.6 web interface hangs long time. style not helpful me. not know why hangs ? ) - i using bash script(as mentioned bellow) , writing time in file( finishtime). problem in additional delay of few second coming , time not giving correct results.
#!/bin/bash e=$(date +%s%n | cut -b1-13) a=($(mapred job -list |awk '{print $1," ",$2," ",$3}' | sort -k1 -n | tail -n 3 | head -n 1)) wait >finishtime destdir=./finishtime >destdir if [ "${a[1]}" = "succeeded" ] total=$(($e - ${a[2]})) echo "$total" > "$destdir" else echo $(($(sed -n '1p' < ./init_setting) * 1000)) > "$destdir" fi exit
- modification in job code , printing time on console( source code of job not available).
if know way of extracting elapsed time of job finished please notify me. if can extract via console.
do know hadoop command "history", can use command print details information of completed job.
the usage "bin/hadoop job -history ".“” on hdfs, job dir. e.g.
bin/hadoop job -history hdfs://172.16.19.5:9000/output/ >> test
note replace path yours
the result on picture
Comments
Post a Comment