https://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
https://www.slideshare.net/BartVandewoestyne/big-data-benchmarking
1、测试云服务器的相关信息及IO
2、测试HDFS的IO
测试云服务器的相关信息及IO
通过测试得到以下信息：
======================================================================== BYTE UNIX Benchmarks (Version 5.1.2) System: host1.cloudlocal.test: GNU/Linux OS: GNU/Linux -- 4.9.75-29.el7.x86_64 -- #1 SMP Fri Jan 5 19:42:28 UTC 2018 Machine: x86_64 (x86_64) Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8") CPU 0: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 1: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 2: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 3: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 4: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 5: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 6: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 7: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 8: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 9: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 10: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 11: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 12: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 13: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 14: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization CPU 15: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips) Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization 06:53:38 up 15:10, 2 users, load average: 0.43, 1.03, 0.91; runlevel 5 ------------------------------------------------------------------------ Benchmark Run: 三 4月 25 2018 06:53:38 - 07:22:16 16 CPUs in system; running 1 parallel copy of tests Dhrystone 2 using register variables 33841196.1 lps (10.0 s, 7 samples) Double-Precision Whetstone 4518.3 MWIPS (10.7 s, 7 samples) Execl Throughput 2049.1 lps (29.8 s, 2 samples) File Copy 1024 bufsize 2000 maxblocks 764894.1 KBps (30.0 s, 2 samples) File Copy 256 bufsize 500 maxblocks 204490.0 KBps (30.0 s, 2 samples) File Copy 4096 bufsize 8000 maxblocks 2521457.9 KBps (30.0 s, 2 samples) Pipe Throughput 1162324.8 lps (10.0 s, 7 samples) Pipe-based Context Switching 139672.3 lps (10.0 s, 7 samples) Process Creation 6805.2 lps (30.0 s, 2 samples) Shell Scripts (1 concurrent) 5691.3 lpm (60.0 s, 2 samples) Shell Scripts (8 concurrent) 976.6 lpm (60.0 s, 2 samples) System Call Overhead 995472.1 lps (10.0 s, 7 samples) System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 33841196.1 2899.8 Double-Precision Whetstone 55.0 4518.3 821.5 Execl Throughput 43.0 2049.1 476.5 File Copy 1024 bufsize 2000 maxblocks 3960.0 764894.1 1931.6 File Copy 256 bufsize 500 maxblocks 1655.0 204490.0 1235.6 File Copy 4096 bufsize 8000 maxblocks 5800.0 2521457.9 4347.3 Pipe Throughput 12440.0 1162324.8 934.3 Pipe-based Context Switching 4000.0 139672.3 349.2 Process Creation 126.0 6805.2 540.1 Shell Scripts (1 concurrent) 42.4 5691.3 1342.3 Shell Scripts (8 concurrent) 6.0 976.6 1627.6 System Call Overhead 15000.0 995472.1 663.6 ======== System Benchmarks Index Score 1096.1 ------------------------------------------------------------------------ Benchmark Run: 三 4月 25 2018 07:22:16 - 07:50:46 16 CPUs in system; running 16 parallel copies of tests Dhrystone 2 using register variables 251068998.6 lps (10.0 s, 7 samples) Double-Precision Whetstone 55154.7 MWIPS (9.9 s, 7 samples) Execl Throughput 30795.9 lps (29.8 s, 2 samples) File Copy 1024 bufsize 2000 maxblocks 838158.5 KBps (30.0 s, 2 samples) File Copy 256 bufsize 500 maxblocks 214567.8 KBps (30.0 s, 2 samples) File Copy 4096 bufsize 8000 maxblocks 2752367.1 KBps (30.0 s, 2 samples) Pipe Throughput 9622691.1 lps (10.0 s, 7 samples) Pipe-based Context Switching 1170824.0 lps (10.0 s, 7 samples) Process Creation 61039.3 lps (30.0 s, 2 samples) Shell Scripts (1 concurrent) 28312.5 lpm (60.0 s, 2 samples) Shell Scripts (8 concurrent) 5327.4 lpm (60.1 s, 2 samples) System Call Overhead 6848982.3 lps (10.0 s, 7 samples) System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 251068998.6 21514.1 Double-Precision Whetstone 55.0 55154.7 10028.1 Execl Throughput 43.0 30795.9 7161.8 File Copy 1024 bufsize 2000 maxblocks 3960.0 838158.5 2116.6 File Copy 256 bufsize 500 maxblocks 1655.0 214567.8 1296.5 File Copy 4096 bufsize 8000 maxblocks 5800.0 2752367.1 4745.5 Pipe Throughput 12440.0 9622691.1 7735.3 Pipe-based Context Switching 4000.0 1170824.0 2927.1 Process Creation 126.0 61039.3 4844.4 Shell Scripts (1 concurrent) 42.4 28312.5 6677.5 Shell Scripts (8 concurrent) 6.0 5327.4 8879.0 System Call Overhead 15000.0 6848982.3 4566.0 ======== System Benchmarks Index Score 5387.1测试详解：
测试项目项目说明基准线Dhrystone 2 using register variables测试处理字符串116700.0lpsDouble-Precision Whetstone测试浮点数操作的速度和效率55.0MWIPSExecl Throughput此测试考察每秒钟可以执行的 execl 系统调用的次数43.0lpsFile Copy 1024 bufsize 2000 maxblocks测试从一个文件向另外一个文件传输数据的速率3960.0KBpsFile Copy 256 bufsize 500 maxblocks测试从一个文件向另外一个文件传输数据的速率。1655.0KBpsFile Read 4096 bufsize 8000 maxblocks测试从一个文件向另外一个文件传输数据的速率。5800.0KBpsPipe-based Context Switching测试两个进程（每秒钟）通过一个管道交换一个不断增长的整数的次数12440.0lpsPipe Throughput一秒钟内一个进程可以向一个管道写 512 字节数据然后再读回的次数4000.0lpsProcess Creation测试每秒钟一个进程可以创建子进程然后收回子进程的次数（子进程一定立即退出）。126.0lpsShell Scripts (8 concurrent)测试一秒钟内一个进程可以并发地开始一个shell 脚本的 n 个拷贝的次数，n 一般取值1，2，4，8.42.4lpmSystem Call Overhead测试进入和离开操作系统内核的代价，即一次系统调用的代价。6.0lpm通过测试得到以下信息：
CPU Model : Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz CPU Cores : 4 Cores @ 2294.688 MHz x86_64 CPU Cache : 51200 KB OS : Ubuntu 16.04.4 LTS (64 Bit) Hyper-V Kernel : 4.15.0-1013-azure Total Space : 1167.0 GB (25.3 GB Used) Total RAM : 28114 MB (7162 MB Used 12236 MB Buff) Total SWAP : 7167 MB (0 MB Used) Uptime : 0 days 4 hour 32 min Load average : 2.24, 1.75, 1.62 ASN & ISP : AS8075, Microsoft Corporation Organization : Microsoft Azure Location : Virginia, United States / US Region : Virginia ---------------------------------------------------------------------- I/O Speed( 1.0GB ) : 52.7 MB/s I/O Speed( 1.0GB ) : 48.0 MB/s I/O Speed( 1.0GB ) : 45.8 MB/s Average I/O Speed : 48.8 MB/s通过测试得到磁盘挂载信息：
sshuser@hn0-sinocb:~$ df -TH Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 15G 0 15G 0% /dev tmpfs tmpfs 3.0G 302M 2.7G 11% /run /dev/sda1 ext4 1.1T 19G 1.1T 2% / tmpfs tmpfs 15G 25k 15G 1% /dev/shm tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs tmpfs 15G 0 15G 0% /sys/fs/cgroup /dev/sdb1 ext4 212G 7.8G 193G 4% /mnt tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/2008 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/202 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/2007 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/2002 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/203 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/2001 tmpfs tmpfs 3.0G 0 3.0G 0% /run/user/2019利用hdparm测试read的速度：
sshuser@hn0-sinocb:~$ sudo hdparm -t /dev/sda1 /dev/sda1: Timing buffered disk reads: 124 MB in 3.07 seconds = 40.35 MB/sec sshuser@hn0-sinocb:~$ sudo hdparm -t /dev/sdb1 /dev/sdb1: Timing buffered disk reads: 7854 MB in 3.00 seconds = 2617.24 MB/sec利用dd测试服务器自身IO：
# write time dd if=/dev/zero of=/var/test bs=8k count=1000000 # read time dd if=/var/test of=/dev/null bs=8k测试HDFS的IO
HDFS 使用GET、PUT命令测试：
test_io_0的大小为2GB，耗时为44.1s，速度为46.4MB/s sshuser@hn0-sinocb:~$ time hadoop fs -get /benchmarks/TestDFSIO/io_data/test_io_0 /tmp real 0m44.108s user 0m18.905s sys 0m4.411s test_io_0的大小为2GB，耗时为10.9s，速度为187.8MB/s sshuser@hn0-sinocb:~$ time hadoop fs -put /tmp/test_io_0 /tmp real 0m10.910s user 0m20.676s sys 0m3.023sHDFS 使用TestDFSIO命令测试：
通过实验得出：有4个值对IO速度有影响分别是nrFiles、fileSize、replication、Blocksize。
# 写测试 速度为57MB/s sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128M 18/08/03 08:01:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 18/08/03 08:01:44 INFO fs.TestDFSIO: Date & time: Fri Aug 03 08:01:44 UTC 2018 18/08/03 08:01:44 INFO fs.TestDFSIO: Number of files: 10 18/08/03 08:01:44 INFO fs.TestDFSIO: Total MBytes processed: 1280.0 18/08/03 08:01:44 INFO fs.TestDFSIO: Throughput mb/sec: 56.02486103208299 18/08/03 08:01:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 57.28535079956055 18/08/03 08:01:44 INFO fs.TestDFSIO: IO rate std deviation: 9.138790402525261 18/08/03 08:01:44 INFO fs.TestDFSIO: Test exec time sec: 27.783 18/08/03 04:23:22 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 18/08/03 04:23:22 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:23:22 UTC 2018 18/08/03 04:23:22 INFO fs.TestDFSIO: Number of files: 10 18/08/03 04:23:22 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 18/08/03 04:23:22 INFO fs.TestDFSIO: Throughput mb/sec: 44.43027072526776 18/08/03 04:23:22 INFO fs.TestDFSIO: Average IO rate mb/sec: 44.44496536254883 18/08/03 04:23:22 INFO fs.TestDFSIO: IO rate std deviation: 0.818583127053945 18/08/03 04:23:22 INFO fs.TestDFSIO: Test exec time sec: 112.405 18/08/03 04:25:14 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 18/08/03 04:25:14 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:25:14 UTC 2018 18/08/03 04:25:14 INFO fs.TestDFSIO: Number of files: 10 18/08/03 04:25:14 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 18/08/03 04:25:14 INFO fs.TestDFSIO: Throughput mb/sec: 45.32076390271969 18/08/03 04:25:14 INFO fs.TestDFSIO: Average IO rate mb/sec: 45.32685089111328 18/08/03 04:25:14 INFO fs.TestDFSIO: IO rate std deviation: 0.5280477142580408 18/08/03 04:25:14 INFO fs.TestDFSIO: Test exec time sec: 83.44 # 读测试 速度为46MB/s sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 1 -fileSize 1280M 18/08/03 08:09:02 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 18/08/03 08:09:02 INFO fs.TestDFSIO: Date & time: Fri Aug 03 08:09:02 UTC 2018 18/08/03 08:09:02 INFO fs.TestDFSIO: Number of files: 10 18/08/03 08:09:02 INFO fs.TestDFSIO: Total MBytes processed: 1280.0 18/08/03 08:09:02 INFO fs.TestDFSIO: Throughput mb/sec: 46.53868528214078 18/08/03 08:09:02 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.76382827758789 18/08/03 08:09:02 INFO fs.TestDFSIO: IO rate std deviation: 3.126925903426747 18/08/03 08:09:02 INFO fs.TestDFSIO: Test exec time sec: 28.243 18/08/03 04:27:20 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 18/08/03 04:27:20 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:27:20 UTC 2018 18/08/03 04:27:20 INFO fs.TestDFSIO: Number of files: 10 18/08/03 04:27:20 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 18/08/03 04:27:20 INFO fs.TestDFSIO: Throughput mb/sec: 46.67423299740648 18/08/03 04:27:20 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.69725799560547 18/08/03 04:27:20 INFO fs.TestDFSIO: IO rate std deviation: 1.038977669688391 18/08/03 04:27:20 INFO fs.TestDFSIO: Test exec time sec: 68.322 18/08/03 04:29:00 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 18/08/03 04:29:00 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:29:00 UTC 2018 18/08/03 04:29:00 INFO fs.TestDFSIO: Number of files: 10 18/08/03 04:29:00 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 18/08/03 04:29:00 INFO fs.TestDFSIO: Throughput mb/sec: 46.54725296204626 18/08/03 04:29:00 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.61547088623047 18/08/03 04:29:00 INFO fs.TestDFSIO: IO rate std deviation: 1.8338779757659875 18/08/03 04:29:00 INFO fs.TestDFSIO: Test exec time sec: 68.055HDFS 使用TeraSort
# 生成1280M数据，耗时22s sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen 12800000 /tera File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=320258 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 WASB: Number of bytes read=252 WASB: Number of bytes written=1280000000 WASB: Number of read operations=0 WASB: Number of large read operations=0 WASB: Number of write operations=0 Job Counters Launched map tasks=2 Other local map tasks=2 Total time spent by all maps in occupied slots (ms)=21315 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=21315 Total vcore-milliseconds taken by all map tasks=21315 Total megabyte-milliseconds taken by all map tasks=32739840 Map-Reduce Framework Map input records=12800000 Map output records=12800000 Input split bytes=167 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=882 CPU time spent (ms)=28800 Physical memory (bytes) snapshot=2392748032 Virtual memory (bytes) snapshot=5661786112 Total committed heap usage (bytes)=2059927552 org.apache.hadoop.examples.terasort.TeraGen$Counters CHECKSUM=27488073979810235 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=1280000000 # 整理1280M数据，耗时66s sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.6.5.8-7.jar terasort /tera /tmp/tera_rp File System Counters FILE: Number of bytes read=2447891722 FILE: Number of bytes written=3779899611 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 WASB: Number of bytes read=1280263628 WASB: Number of bytes written=1280000000 WASB: Number of read operations=0 WASB: Number of large read operations=0 WASB: Number of write operations=0 Job Counters Launched map tasks=4 Launched reduce tasks=1 Rack-local map tasks=4 Total time spent by all maps in occupied slots (ms)=84380 Total time spent by all reduces in occupied slots (ms)=41598 Total time spent by all map tasks (ms)=84380 Total time spent by all reduce tasks (ms)=41598 Total vcore-milliseconds taken by all map tasks=84380 Total vcore-milliseconds taken by all reduce tasks=41598 Total megabyte-milliseconds taken by all map tasks=129607680 Total megabyte-milliseconds taken by all reduce tasks=63894528 Map-Reduce Framework Map input records=12800000 Map output records=12800000 Map output bytes=1305600000 Map output materialized bytes=1331200024 Input split bytes=664 Combine input records=0 Combine output records=0 Reduce input groups=12800000 Reduce shuffle bytes=1331200024 Reduce input records=12800000 Reduce output records=12800000 Spilled Records=36337420 Shuffled Maps =4 Failed Shuffles=0 Merged Map outputs=4 GC time elapsed (ms)=1500 CPU time spent (ms)=97490 Physical memory (bytes) snapshot=4674580480 Virtual memory (bytes) snapshot=14154989568 Total committed heap usage (bytes)=5284298752 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1280261968 File Output Format Counters Bytes Written=1280000000 # Tera的Teravalidate，耗时53s sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.6.5.8-7.jar teravalidate /tera /tmp/report File System Counters FILE: Number of bytes read=672045548 FILE: Number of bytes written=1344572754 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 WASB: Number of bytes read=1280000498 WASB: Number of bytes written=659244666 WASB: Number of read operations=0 WASB: Number of large read operations=0 WASB: Number of write operations=0 Job Counters Launched map tasks=2 Launched reduce tasks=1 Rack-local map tasks=2 Total time spent by all maps in occupied slots (ms)=53714 Total time spent by all reduces in occupied slots (ms)=13983 Total time spent by all map tasks (ms)=53714 Total time spent by all reduce tasks (ms)=13983 Total vcore-milliseconds taken by all map tasks=53714 Total vcore-milliseconds taken by all reduce tasks=13983 Total megabyte-milliseconds taken by all map tasks=82504704 Total megabyte-milliseconds taken by all reduce tasks=21477888 Map-Reduce Framework Map input records=12800000 Map output records=6400438 Map output bytes=659244660 Map output materialized bytes=672045548 Input split bytes=332 Combine input records=0 Combine output records=0 Reduce input groups=6 Reduce shuffle bytes=672045548 Reduce input records=6400438 Reduce output records=6400434 Spilled Records=12800876 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=1680 CPU time spent (ms)=52150 Physical memory (bytes) snapshot=2986717184 Virtual memory (bytes) snapshot=8509726720 Total committed heap usage (bytes)=3184525312 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1280000000 File Output Format Counters Bytes Written=659244666Ambari的HDFS测试在当前配置下，IO速度为50MB/s。
