Dell PowerEdge C5220 Bedienungsanleitung

A PRINCIPLED TECHNOLOGIES TEST REPORT

Commissioned by Dell Inc.; April 2012

DELL POWEREDGE C5220: HADOOP MAPREDUCE PERFORMANCE

Every year, the amount of data that businesses must process grows

enormously. The ability to sort, filter, and analyze this data is becoming more

and more vital to many businesses in analyzing their customers and their market

segment. Additionally, businesses need an infrastructure that is powerful and

flexible, but also compact and scale-friendly. The Dell PowerEdge C5220 server is

an ideal solution to pair with Apache Hadoop, a powerful multi-node data

analysis application. With the PowerEdge C5220, organizations can scale out to

their data processing requirements and successfully handle these ever-increasing

data volumes, finding new value in their big data.

To test the Hadoop performance capabilities of the Dell PowerEdge

C5220, we configured eight Dell PowerEdge C5220 servers into a Hadoop cluster

and ran the MapReduce benchmark (mrbench) on the platform. We found that

eight Dell PowerEdge C5220 servers, all contained within the single shared

infrastructure design of the Dell PowerEdge C5000 chassis, could run our

mrbench tests of varying sizes, map processes, and reduce processes, in times

averaging just 15.9 to 25.6 seconds, making this platform ideal for scale-out

data-analysis application workloads.

A Principled Technologies test report 2

Dell PowerEdge C5220: Hadoop MapReduce Performance

DELL POWEREDGE C5220 AND HADOOP –BETTER TOGETHER

Today’s marketplace has moved beyond the structural confines of relational

data and up to the category of “big data.” Companies with hyperscale data centers are

dealing with tremendous amounts of existing data, with massive data growth on the

horizon. They require a reliable and powerful hardware platform, along with a reliable

and fast software platform on which to process this big data. The Dell PowerEdge C

series servers provide this solid infrastructure for companies to offer their data

processing capabilities. Integrating many PowerEdge C servers together in a Hadoop

cluster allows these companies to quickly and efficiently process large datasets.

In our lab tests, we configured eight Dell PowerEdge C5220 microservers into a

Hadoop cluster and ran the MapReduce benchmark on the cluster to demonstrate the

capabilities of the Dell PowerEdge C platform. Below we discuss the Dell PowerEdge

C5220 microserver, our test topology, and our results in more detail.

Figure 1: The Dell PowerEdge

C5220 microserver provides

power and reliability in a tight

and dense form factor, within

the C5000 shared

infrastructure.

Dell PowerEdge C5220

The Dell PowerEdge C5220

microserver is a highly dense small-

form-factor server mounted on a sled,

which is housed in a Dell PowerEdge

C5000 shared infrastructure enclosure,

offering up to 12 microservers in a 3U

rack density. These high physical

densities offer massive scale-out

configurations, and are ideal for service

providers and large data centers that

need optimized space, power, and

cooling infrastructure. Each server, all

cold-aisle accessible for maintenance,

features processing power from the Intel® Xeon® E3-1200 series, up to 32GB RAM

Highlights of the Dell

PowerEdge C5220

Up to 12 PowerEdge C5220 microservers in a

single 3U C5000 chassis

Cold aisle serviceability

Shared infrastructure uses less floor space,

power, and cooling

Dense form factor ideal for service providers,

hosting platforms, and hyperscale

environments

Up to 6TB raw storage in a single

microserverEmbedded chassis management

control

A Principled Technologies test report 3

Dell PowerEdge C5220: Hadoop MapReduce Performance

capacity, and up to four drives per server. The high performance levels it delivers,

coupled with a simple, modular, and easy-to-maintain design, make the PowerEdge

C5220 microserver an optimal choice to deploy for extremely dense compute fabrics

handling big data deployments, large software as a service (SaaS) environments, and

cloud deployments. Figure 2 presents a view of the Dell PowerEdge C5220.

Figure 2: The PowerEdge C5220 microserver.

The Dell PowerEdge C5220 can deliver many benefits to your large cloud

deployment, and lets you:

Maintain your environment easily with cold-aisle accessible, swappable

microserver modules and connections.

Make better use of expensive data center space by increasing the rack

density with up to twelve server nodes fitting into a standard 3U rack slot.

Utilize up to 6 TB of raw disk storage per microserver with SAS drives or 4 TB

of raw disk storage per microserver with SATA drives –up to 72 TB and 48

TB respectively –across the entire C5000 shared infrastructure in just a 3U

form factor

A Principled Technologies test report 4

Dell PowerEdge C5220: Hadoop MapReduce Performance

Reuse or repurpose servers easily when workloads change with hot-swap

server nodes –you no longer need to experience downtime by replacing the

entire server chassis.

Designed with power-efficiency and maintainability in, the Dell PowerEdge

C5220 maximizes operating efficiency with a shared-infrastructure design. To learn

more about the Dell PowerEdge C5220 and the entire Dell PowerEdge C Series, visit

http://www.dell.com/us/enterprise/p/poweredge-cloud-servers.

WHAT WE TESTED

To test the ability of the PowerEdge C5220 microserver to handle large data

processing tasks, we used Hadoop, specifically Cloudera Distribution Including Apache

Hadoop (CDH). Below, we briefly discuss Hadoop and the benchmark tool we used,

MapReduce benchmark (mrbench).

Hadoop

Hadoop, developed by Apache Software Foundation, is an open-source

distributed application that enables the analysis of large volumes of data for specific

purposes. Using Hadoop’s framework, IT organizations and researchers can build

applications that tailor the data analysis to specific needs for each company, even using

unstructured data. Many different markets—among them finance, IT, and retail—use

Hadoop due to its ability to handle heterogeneous data, both structured and

unstructured.

Hadoop can run across any number of machines using varied hardware,

spreading data across all available hardware resources using a distributed file system,

Hadoop Distributed File System (HDFS), and replicating data to minimize loss if a

hardware malfunction occurs. The software is able to detect hardware failures, and to

work around said failures to allow uninterrupted access to data. Because of its ability to

run on different hardware, a Hadoop cluster is scalable and flexible –it can be expanded

to encompass growing databases and companies. It is also cost-effective as it allows

companies to utilize commodity hardware effectively.

MapReduce

MapReduce is a framework within Hadoop that provides the ability to process

extremely large datasets in parallel across the Hadoop cluster, shortening the overall

processing time greatly. MapReduce breaks input data down into chunks to be

processed across the Hadoop cluster. When an application is run on a Hadoop cluster,

MapReduce perfoms “map” tasks that process data in parallel. The data is then sent to

“reduce” tasks that reduce the information into a final result. This allows for faster data

processing using multiple nodes, while still producing a single, comprehensive, accurate

result.

A Principled Technologies test report 5

Dell PowerEdge C5220: Hadoop MapReduce Performance

TEST RESULTS IN MORE DETAIL

We used mrbench to create datasets containing varying numbers of records,

which each contained 6 bytes, for datasets varying in size from 96 bytes to 9,600,000

bytes. We tested two mrbench configurations; in the first, we applied 16 maps and one

reduction process to those input recordsets, and in the second, we applied 160 maps

and 16 reduction processes to those input recordsets.

Figure 4 shows the time in milliseconds the cluster took to complete each job

when we applied 16 maps and one reduction process to those input recordsets.

Task

Number of records

Total bytes

Time to complete (milliseconds)

Task 1

14,955

Task 2

160

960

14,559

Task 3

1,600

9,600

14,515

Task 4

16,000

96,000

14,851

Task 5

160,000

960,000

14,846

Task 6

1,600,000

9,600,000

15,878

Figure 3: The time spent for each job of the Hadoop MapReduce benchmark.

Figure 4 shows the time in milliseconds the cluster took to complete each job

when we applied 160 maps and 16 reduction processes to those input recordsets.

Task

Number of records

Total bytes

Time to complete (milliseconds)

Task 1

22,374

Task 2

160

960

22,057

Task 3

1,600

9,600

22,496

Task 4

16,000

96,000

21,648

Task 5

160,000

960,000

22,122

Task 6

1,600,000

9,600,000

25,594

Figure 4: The time spent for each job of the Hadoop MapReduce benchmark.

CONCLUSION

Companies that wish to process big data in scale-out environments require

powerful and dense hardware that complete large data processing tasks, such as

Hadoop analysis, quickly and efficiently. Selecting the right server for your underlying

hardware infrastructure is critical at hyperscale.

In our tests, an eight-node Dell PowerEdge C5220 Hadoop cluster was able to

efficiently process multiple MapReduce scenarios. While each scenario and each

environment vary, our eight-node Hadoop MapReduce test shows that even a small

number of Dell PowerEdge C5220 microservers can process varying amounts of data

quickly.

A Principled Technologies test report 6

Dell PowerEdge C5220: Hadoop MapReduce Performance

APPENDIX A –SERVER CONFIGURATION INFORMATION

Figure 5 provides detailed configuration information for the test servers.

System

Dell PowerEdge C5220

General

Number of processor packages

Number of cores per processor

Number of hardware threads per core

System power management policy

Balanced

CPU

Vendor

Intel

Name

Xeon

Model number

E3-1220

Stepping

Socket type

LGA1155

Core frequency (GHz)

3.20

Bus frequency

5 GT/s

L1 cache

32 KB + 32 KB (per core)

L2 cache

256 KB (per core)

L3 cache (MB)

Platform

Vendor and model number

Dell PowerEdge C5220

Motherboard model number

1KTH4B0

BIOS name and version

Dell 1.0.11

BIOS settings

Defaults

Memory module(s)

Total RAM in system (GB)

Vendor and model number

Samsung M391B1G73AH0-YH9

Type

PC3L-10600E

Speed (MHz)

1,333

Speed running in the system (MHz)

1,333

Timing/Latency (tCL-tRCD-tRP-tRASmin)

9-9-9-36

Size (GB)

Number of RAM module(s)

Chip organization

Double-sided

Rank

Dual

Operating system

Name

CentOS 6.2, x86_64

File system

ext4

Kernel

2.6.32-220.13.1.el6.x86_64

Language

English

Updates

All as of 4/12/2012

Graphics

Vendor and model number

AST2050

A Principled Technologies test report 7

Dell PowerEdge C5220: Hadoop MapReduce Performance

System

Dell PowerEdge C5220

Graphics memory (MB)

128

RAID controller

Vendor and model number

Intel C204 SATA Raid Controller

Driver version

N/A

Cache size (GB)

None

Hard drive

Vendor and model number

Dell ST9500620NS

Number of disks in system

Size (GB)

500

Buffer size (MB)

RPM

7.2K

Type

SATA

Ethernet adapters

Vendor and model number

Intel 82580DB Gigabit Network Connection

Type

Integrated

Driver

Intel(R) Gigabit Ethernet Network Driver; igb, 3.0.6-k

USB ports

Number

1 internal

Type

2.0

Figure 5: Configuration details for the two test servers.

A Principled Technologies test report 8

Dell PowerEdge C5220: Hadoop MapReduce Performance

APPENDIX B –HOW WE TESTED

We installed and configured CentOS on an auxiliary server, which provided both basic services and Hadoop

management for the C5220 nodes in the Hadoop cluster. We installed and configured CentOS on the C5220 nodes, using

the C5000’s baseboard management controller (BMC) virtual console feature. We installed the Cloudera Hadoop

Manager on the auxiliary manager server and used it to install Hadoop on the C5220 nodes.

Configuring the auxiliary server

Below, we review the steps necessary to configure the management server, which runs the necessary auxiliary

services and the Cloudera Manager.

1. Install the base-installation of CentOS 6.2 x86_64 following the steps in the section CentOS base installation.

2. Log onto the server as root.

3. Configure one network interface, e.g. eth0, to access a network connected to the Internet with, for example, IP

address 10.1.1.10.

4. Configure one network interface, e.g., eth1, to access the network of C5220 nodes with, for example, IP address

192.168.1.10 and CIDR prefix of 24.

5. Install CentOS updates:

yum update

6. Reboot the server:

shutdown –r now

7. After the system reboots, log onto the server as root.

8. Install Squid as a HTTP proxy for the C5220 nodes:

yum install squid

9. Modify /etc/squid/squid.conf configuration to add these settings:

acl ftp proto FTP

http_access allow ftp

## cluster network

acl intranet src 192.168.1.0/21

http_access allow intranet

10. Install dnsmasq to supply hostname resolution for the nodes:

yum install dnsmasq

11. Edit the configuration /etc/dnsmasq.conf and modify these lines as shown:

interface=eth1

no-dhcp-interface=eth1

12. Add the IP addresses and hostnames of the auxiliary server and C5220 nodes to /etc/hosts on the auxiliary

server. For example:

192.168.1.10 had01-ctrl01

192.168.1.31 cl01n01

192.168.1.32 cl01n02

192.168.1.33 cl01n03

192.168.1.34 cl01n04

192.168.1.35 cl01n05

192.168.1.36 cl01n06

192.168.1.37 cl01n07

192.168.1.38 cl01n08

13. Start the DNS server:

chkconfig dnsmasq on

A Principled Technologies test report 9

Dell PowerEdge C5220: Hadoop MapReduce Performance

service dnsmasq start

14. Install the ntpd time server to provide a common time source for the nodes:

yum install ntpd

15. Modify the configuration file /etc/ntp.conf to use Internet time sources, and to be a time service on the nodes

network. For example:

# Hosts on local network are less restricted.

restrict 192.168.1.0 mask 255.255.255.0

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

server 0.pool.ntp.org

16. Start the NTP server:

chkconfig ntpd on

service ntpd start

17. Generate an SSH public-key pair with, for example, no password:

ssh-keygen -t dsa -N ""

18. Reboot the server.

Configuring the C5220 Hadoop nodes

1. Install the base-installation of CentOS 6.2 x86_64 following the steps in the section CentOS base installation.

2. Configure one network interface, e.g. em1, to access the Hadoop network of C5220 nodes with, for example, IP

address 192.168.1.31 and CIDR prefix of 24.

3. Configure the system to synchronize its time with the NTP service on the auxiliary server. Replace the server

lines in /etc/ntp.conf with this:

server 192.168.1.10

4. Configure the yum updater to use the cluster’s HTTP proxy:

echo "proxy=http://192.168.1.10:3128/" >> /etc/yum.conf

5. Copy the auxiliary server’s public SSH key to the node to allow easy remote management:

mkdir /root/.ssh

chmod 700 /root/.ssh

scp 192.168.1.10:/root/.ssh/id_dsa.pub /root/.ssh/authorized_keys

6. Install CentOS updates:

yum update

7. Reboot the server:

shutdown –r now

8. Repeat steps 1 through 7 for the remaining nodes.

Cloudera Hadoop (CDH3u3) installation using Cloudera manager (3.7.5)

1. Log onto the auxiliary server as root.

2. Download and transfer the free edition of the Cloudera Manager installer, version 3.7.5, for CentOS from

Cloudera.com to the auxiliary server.

3. Run the Cloudera Manager installer:

./cloudera-manager-installer.bin

4. On the Cloudera Manager README screen, read the installation overview, and select Next.

5. On the Cloudera Manager (Free Edition) License screen, review the license, and select Next.

6. On the next screen, select yes to accept this license.

7. On the Oracle Binary Code License Agreement screen, review the license, and select Next.

A Principled Technologies test report 10

Dell PowerEdge C5220: Hadoop MapReduce Performance

8. On the next screen, select Yes to accept this license.

9. The auxiliary server now has a Web interface for managing Hadoop. Select OK on the Next Step screen.

10. On the installer’s Finish screen, select OK to exit the installer.

11. Using a Web browser, open the page http://<clouderamanager>:7180/ , where <clouderamanager> is the IP

address of the auxiliary manager server.

12. Copy the SSH keys, previously generated in the Configuring the auxiliary server section, from the auxiliary server

to the local desktop running the Web browser.

13. Log onto the Cloudera manager with username admin and password admin.

14. On the Thank you for choosing Cloudera Manager and Cloudera’s Distribution including Apache Hadoop (CDH)

screen, click Continue.

15. On the register your CDH3 installation, click Skip Registration.

16. On the Specify hosts for your CDH3 cluster installation screen, enter a list of hostnames or IP addresses in the

text box at the bottom, such as 192.168.1.3[1-8]. Click Find Hosts.

17. The upper text box will contain a list of potential Hadoop hosts. Select the applicable nodes, and click Continue.

18. On the provide SSH login credentials screen, select All hosts use the same public key, click Choose file for the

Public Key File, and browse to and select the SSH public-key file for the auxiliary server’s public key (see step 12).

19. Repeat step 18 to select the auxiliary server’s Private Key File.

20. Click Start Installation.

21. After the Hadoop installation on the nodes has finished, switch to a console session on the auxiliary server.

22. Format two of the remaining four disks on each node as part of the Hadoop file system.

# Note: in the following example, the contents of disks /dev/sdc and /dev/sdd

# will be destroyed

# partition /dev/sdc and /dev/sdd and create EXT4 file systems

for i in $(seq 31 38); do

ssh 192.168.1.$i parted -s /dev/sdc mklab gpt \; parted -s /dev/sdd mklab gpt

ssh 192.168.1.$i parted /dev/sdc mkpart primary '"1 -1"' \; \

parted /dev/sdd mkpart primary '"1 -1"'

ssh 192.168.1.$i mkfs.ext4 /dev/sdc1 \; mkfs.ext4 /dev/sdd1

done

# Modify fstab so that these file systems are mounted at boot time

for i in $(seq 31 38); do

ssh 192.168.1.$i '(echo "/dev/sdc1 /dfs/d1 ext4 defaults,noatime 1 2";\

echo "/dev/sdd1 /dfs/d2 ext4 defaults,noatime 1 2") \

>> /etc/fstab'

done

# Create the default Hadoop HDFS directories

for i in $(seq 31 38); do

ssh 192.168.1.$i mkdir -p '/dfs/{d1,d2,n1,n2,s1,s2}' '/mapred/{local,jt}'

ssh 192.168.1.$i mount '/dfs/d{1,2}'

done

for i in $(seq 31 38); do

ssh 192.168.1.$i chmod 700 '/dfs/{d1,d2,n1,n2,s1,s2}' \;

chmod 755 '/mapred/{local,jt}'

Andere Handbücher für PowerEdge C5220

Inhaltsverzeichnis

Andere Dell Server Handbücher

Dell

Dell PowerEdge 2410 Bedienungsanleitung

Dell

Dell PowerEdge R530xd Bedienungsanleitung

Dell

Dell PowerEdge R715 Bedienungsanleitung

Dell

Dell PowerEdge systems 6300 Service-Handbuch

Dell

Dell ECS EX Series Montageanleitung

Dell

Dell POWEREDGE R710 Bedienungsanleitung

Dell

Dell PowerEdge 2950 Bedienungsanleitung

Dell

Dell E60S Handbuch

Dell

Dell PowerEdge C5220 Handbuch

Dell

Dell 5J5D0 Handbuch

Dell

Dell PowerEdge 1750 Bedienungsanleitung

Dell

Dell E665 Bedienungsanleitung

Dell

Dell PowerEdge 1950 Bedienungsanleitung

Dell

Dell PS4000E Bedienungsanleitung

Dell

Dell PowerEdge C5125 Bedienungsanleitung

Dell

Dell PowerEdge R210 Bedienungsanleitung

Dell

Dell PowerEdge C6105 Bedienungsanleitung

Dell

Dell PowerEdge C6320 Bedienungsanleitung

Dell

Dell PowerEdge C6220 Bedienungsanleitung

Dell

Dell poweredge VRTX Bedienungsanleitung

Dell

Dell DX6004S Bedienungsanleitung

Dell

Dell Force10 S2410-01-10GE-24P Installationshandbuch

Dell

Dell PowerEdge 6350 Bedienungsanleitung

Dell

Dell PowerEdge R650XS M4JNT Handbuch

Beliebte Server Handbücher anderer Marken

HP AlphaServer ES47 Bedienungsanleitung

PowerNAS

PowerNAS CMA Bedienungsanleitung

Supero

Supero FatTwin F617R3-FT Bedienungsanleitung

EtroVISION

EtroVISION EV3151 Bedienungsanleitung

HP ProLiant Gen9 Bedienungsanleitung

Compaq

Compaq ProLiant 3000 Handbuch

iRobo

iRobo IPC2U Bedienungsanleitung

Nortel

Nortel 1000 Con?guration guide Bedienungsanleitung

Asus

Asus AP7500 Bedienungs- und Wartungshandbuch

Avid Technology

Avid Technology AirSpeed 5000 Bedienungsanleitung

HP Integrity rx2600 Installationsanleitung

Milestone

Milestone Husky IVO 350T Bedienungsanleitung

Planet

Planet NAS-7410 Bedienungsanleitung

Lenovo

Lenovo ThinkSystem SR635 V3 Bedienungsanleitung

Fujitsu

Fujitsu PRIMERGY BX900 Bedienungsanleitung

Asus

Asus RS700DA-E6 PS4 Bedienungsanleitung

Vertiv

Vertiv Avocent ACS6000 Montage- und Wartungsanleitung

A & T Internationale

A & T Internationale Xserve Service-Handbuch