建站
咨詢
售后
建站咨詢
新聞中心
這里有您想知道的互聯(lián)網(wǎng)營銷解決方案
云服務(wù)器搭建hadoop
在云服務(wù)器中配置Hadoop的過程主要包括以下幾個(gè)步驟：安裝Java環(huán)境、下載并解壓Hadoop安裝包、配置Hadoop環(huán)境變量、初始化Hadoop集群、配置Hadoop的核心配置文件、啟動(dòng)Hadoop集群，下面詳細(xì)介紹每個(gè)步驟的具體操作。
創(chuàng)新互聯(lián)IDC提供業(yè)務(wù):溫江服務(wù)器托管,成都服務(wù)器租用,溫江服務(wù)器托管,重慶服務(wù)器租用等四川省內(nèi)主機(jī)托管與主機(jī)租用業(yè)務(wù);數(shù)據(jù)中心含:雙線機(jī)房,BGP機(jī)房,電信機(jī)房,移動(dòng)機(jī)房,聯(lián)通機(jī)房。
1、安裝Java環(huán)境
Hadoop依賴于Java環(huán)境，因此首先需要在云服務(wù)器上安裝Java，可以使用以下命令安裝OpenJDK:
sudo apt-get update sudo apt-get install openjdk-8-jdk
2、下載并解壓Hadoop安裝包
從Hadoop官網(wǎng)下載最新版本的Hadoop安裝包，然后將其上傳到云服務(wù)器上，接下來使用以下命令解壓安裝包：
tar -xzf hadoop-x.y.z.tar.gz
3、配置Hadoop環(huán)境變量
編輯~/.bashrc文件，添加以下內(nèi)容：
export HADOOP_HOME=/path/to/hadoop-x.y.z export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
保存文件后，執(zhí)行以下命令使配置生效：
source ~/.bashrc
4、初始化Hadoop集群
使用以下命令初始化Hadoop集群：
hadoop namenode -format
5、配置Hadoop的核心配置文件
編輯$HADOOP_CONF_DIR/core-site.xml文件，添加以下內(nèi)容：
fs.defaultFS hdfs://localhost:9000
編輯$HADOOP_CONF_DIR/hdfs-site.xml文件，添加以下內(nèi)容：
dfs.replication 1
6、啟動(dòng)Hadoop集群
使用以下命令啟動(dòng)Hadoop集群：
start-dfs.sh start namenode start-yarn.sh start resourcemanager yarn --config $HADOOP_CONF_DIR/yarn-site.xml start nodemanager --config $HADOOP_CONF_DIR/yarn-site.xml host=localhost port=8042 state=started user=root group=root memory=2048 cpus=1 vcores timeout=120000 queues=default(a-default) +capacity(a-capacity) +userlimit(a-userlimit) +a-fair(a-fair) +placeresume(a-placeresume) +port(a-port) +ip(a-ip) +hostsfile(a-hostsfile) +hostsallow(a-hostsallow) +hostsdeny(a-hostsdeny) +hostsexclude(a-hostsexclude) +topology(a-topology) +acl(a-acl) +loglevel(a-loglevel) +healthcheck(a-healthcheck) +skipcrccheck(a-skipcrccheck) +enableautofailover(a-enableautofailover) +failoverrecoverytime(a-failoverrecoverytime) +maxtaskfailures(a-maxtaskfailures) +tasktrackerexpiryintervalsecs(a-tasktrackerexpiryintervalsecs) +schedulerclasspath(a-schedulerclasspath) +schedulerproperties(a-schedulerproperties) +driverclasspath(a-driverclasspath) +driverproperties(a-driverproperties) +preemptable(a-preemptable) +priorityforhighpritasks(a-priorityforhighpritasks) +maxidletimeoutmins(a-maxidletimeoutmins) +heartbeatdelaysecs(a-heartbeatdelaysecs) +heartbeatmonitorthreads(a-heartbeatmonitorthreads) +clientprogressreporting(a-clientprogressreporting) +unmanageddisksacceptingsnapscans(a-unmanageddisksacceptingsnapscans) +blockadmitrate(a-blockadmitrate) +vadmissionrate(a-vadmissionrate) +replicationminimum(a-replicationminimum) +replicationmaximum(a-replicationmaximum) +saveNamespacestate(a-saveNamespacestate) +saveNamespacestateOnExit(a-saveNamespacestateOnExit) +transitionalCheckpointTimeSecs(a-transitionalCheckpointTimeSecs) +stateSavePeriodic(a-stateSavePeriodic) +safemodeWaitSeconds(a-safemodeWaitSeconds) +ipRange(a-ipRange);;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;+transferAckWindowSize(xfer_ack_window_size);+transferBlockLimitRate(xfer_block_limit_rate);+transferBandwidthPerSec(xfer_bandwidth);+transferBlockSizeBits(xfer_blocksize);+transferProtocol(xfer_protocol);+streamBlockSizeBits(stream_blocksize);+streamBlockCacheSize(stream_blockcachesize);+streamReadaheadSize(streamreadaheadsize);+streamSocketTimeoutSecs(streamsockettimeoutsecs);+dataTransferProtocolClassName(dataXferProtocolClassName);+dataTransferProtocolProxyPortNumber(dataXferProtocolProxyPortNumber);+dataTransferProtocolProxyHostNamePatternString(dataXferProtocolProxyHostNamePatternString);+dataTransferProtocolUsePlainTextModeForStreamingTcpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts(dataXferProtocolUsePlainTextModeForStreamingTcpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts);+dataTransferProtocolUsePlainTextModeForStreamingUdpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts(dataXferProtocolUsePlainTextModeForStreamingUdpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts);+resourceManagerClusterEnabled (resourcemanagerclusterenabled);+resourceManagerClusterTrackingUrl (resourcemanagerclustertrackingurl);+resourceManagerHostname (resourcemanagerhostname);+resourceManagerPortNumber (resourcemanagerportnumber);+resourceManagerHttpPort (resourcemanagerhttpport);+resourceManagerMaxAppSubmissionBatchSize (resourcemanagermaxappsubmissionbatchsize);+resourceManagerMaxAMMemoryMB (resourcemanagermaxammemorymb);+resourceManagerMaxAppAttemptsPerContainerType (resourcemanagermaxappattemptspercontainertype);+resourceManagerMaxRunningAppAttemptCount (resourcemanagermaxrunningappattemptcount);+resourceManagerMaxContainerCapacity (resourcemanagermaxcontainercapacity);+resourceManagerMaxAMContainerSize (resourcemanagermaxamcontainersize);+resourceManagerAsyncAmStartTimeoutSeconds (resourcemanagerasyncamstarttimeoutseconds);+resourceManagerAsyncAmFinishTimeoutSeconds (resourcemanagerasyncamfinishtimeoutseconds);+resourceManagerAsyncAmPollIntervalSeconds (resourcemanagerasyncampollintervalseconds);+yarnNodeManagerHostname (yarnnodemanagerhostname);+yarnNodeManagerBindAddress (yarnnodemanagerbindaddress);+yarnNodeManagerPortNumber (yarnnodemanagerportnumber);+yarnNodeManagerWebInterfacePortNumber (yarnnodemanagerwebinterfaceportnumber);+yarnNodeManagerEnableNodelifecycleMonitoringService (yarnnodemanagerenablenodelifecyclemonitoringservice);+yarnNodeManagerHeapSizeMB (yarnnodemanagerheapsizemb);+yarnNodeManagerCpuLoadBalancingMinimumSystemResourcePercent (yarnnodemanagercpuloadbalancingminimumsystemresourcepercent);+yarnNodeManagerCpuLoadBalancingMaximumSystemResourcePercent (yarnnodemanagercpuloadbalancingmaximumsystemresourcepercent);+yarnNodeManagerCpuLoadBalancingEnabled (yarnnodemanagercpuloadbalancingenabled);+yarnNodeManagerDiskFailureThresholdPercentage (yarnnodemanagerdiskfailurethresholdpercentage);+yarnNodeManagerDiscardReclaimedExportedKeytabEntriesAfterNHours (yarnnodemanagerdiscardreclaimedexportedkeytabentriesafternhhours);+yarnNodeManagerTokenAuthenticatorFile (yarnnodemanagertokenauthenticatorfile);+yarnNodeManagerDelegationTokenExpirationMsec (yarnnodemanagerdelegationtokenexpirationmsec);+yarnNodeManagerClientUserAgentPrefixesWhitelistFileName (yarnnodemanagerclientuseragentprefixeswhitelistfilename);+yarnNodeManagerClientUserAgentSuffixesBlacklistFileName (yarnnodemanagerclientuseragentsuffixesblacklistfilename);+yarnNodeManagerLogLevel (yarnnodemanagerloglevel);+yarnNodeManagerLogDir (yarnnodemanagerlogdir);+oozie
新聞標(biāo)題：云服務(wù)器搭建hadoop
分享鏈接：http://www.dlmjj.cn/article/cddpogp.html
