Course Code

hadoopadm1

Duration

21 hours (usually 3 days including breaks)

Requirements

comfortable with basic Linux system administration
basic scripting skills

Knowledge of Hadoop and Distributed Computing is not required, but will be introduced and explained in the course.

Lab environment

Zero Install : There is no need to install hadoop software on students’ machines! A working hadoop cluster will be provided for students.

Students will need the following

an SSH client (Linux and Mac already have ssh clients, for Windows Putty is recommended)
a browser to access the cluster. We recommend Firefox browser with FoxyProxy extension installed

Overview

Apache Hadoop 은 서버 클러스터에서 Big Data 를 처리하는 데 가장 널리 사용되는 프레임 워크입니다. 에서 세 (선택적 사) 일 물론, 참석자가 비즈니스 혜택과 사용 사례에 대해 배우게됩니다이 Hadoop , 설치, 유지 관리, 모니터링, 문제를 해결하고 최적화하는 방법 클러스터 구축과 성장을 계획하는 방법과 생태계, Hadoop . 또한 클러스터 대량 데이터로드를 연습하고 다양한 Hadoop 배포판에 익숙해지며 Hadoop 에코 시스템 도구 설치 및 관리를 연습합니다. 이 과정은 Kerberos를 사용한 클러스터 보안에 대한 논의로 마무리됩니다.

“… 재료를 잘 준비하고 철저히 덮었습니다. 실험실은 매우 도움이되고 체계적으로 구성되었습니다.”
— Andrew Nguyen, Microsoft 온라인 Advertising 수석 통합 DW 엔지니어

청중

Hadoop 관리자

체재

강의 및 실습, 대략 60 % 강의, 40 % 실험실.

Machine Translated

Course Outline

Introduction
- Hadoop history, concepts
- Ecosystem
- Distributions
- High level architecture
- Hadoop myths
- Hadoop challenges (hardware / software)
- Labs: discuss your Big Data projects and problems

Planning and installation
- Selecting software, Hadoop distributions
- Sizing the cluster, planning for growth
- Selecting hardware and network
- Rack topology
- Installation
- Multi-tenancy
- Directory structure, logs
- Benchmarking
- Labs: cluster install, run performance benchmarks

HDFS operations
- Concepts (horizontal scaling, replication, data locality, rack awareness)
- Nodes and daemons (NameNode, Secondary NameNode, HA Standby NameNode, DataNode)
- Health monitoring
- Command-line and browser-based administration
- Adding storage, replacing defective drives
- Labs: getting familiar with HDFS command lines

Data ingestion
- Flume for logs and other data ingestion into HDFS
- Sqoop for importing from SQL databases to HDFS, as well as exporting back to SQL
- Hadoop data warehousing with Hive
- Copying data between clusters (distcp)
- Using S3 as complementary to HDFS
- Data ingestion best practices and architectures
- Labs: setting up and using Flume, the same for Sqoop

MapReduce operations and administration
- Parallel computing before mapreduce: compare HPC vs Hadoop administration
- MapReduce cluster loads
- Nodes and Daemons (JobTracker, TaskTracker)
- MapReduce UI walk through
- Mapreduce configuration
- Job config
- Optimizing MapReduce
- Fool-proofing MR: what to tell your programmers
- Labs: running MapReduce examples

YARN: new architecture and new capabilities
- YARN design goals and implementation architecture
- New actors: ResourceManager, NodeManager, Application Master
- Installing YARN
- Job scheduling under YARN
- Labs: investigate job scheduling

Advanced topics
- Hardware monitoring
- Cluster monitoring
- Adding and removing servers, upgrading Hadoop
- Backup, recovery and business continuity planning
- Oozie job workflows
- Hadoop high availability (HA)
- Hadoop Federation
- Securing your cluster with Kerberos
- Labs: set up monitoring

Optional tracks
- Cloudera Manager for cluster administration, monitoring, and routine tasks; installation, use. In this track, all exercises and labs are performed within the Cloudera distribution environment (CDH5)
- Ambari for cluster administration, monitoring, and routine tasks; installation, use. In this track, all exercises and labs are performed within the Ambari cluster manager and Hortonworks Data Platform (HDP 2.0)

회원 평가

★★★★★

교육에 대해 가장 좋아하는 점은 무엇입니까? : 각 도구에 대해 자세히 설명합니다.

Amber Mehrotra - NIIT Limited

Course:Hadoop Administration

Machine Translated

트레이너가 외부 디스크에 제공 한 이미 준비된 VM에서 모든 데이터와 소프트웨어를 사용할 준비가되었다는 사실.

vyzVoice

Course:Hadoop for Developers and Administrators

Machine Translated

트레이너가 예를 보여줍니다

Simon Hahn