Category Archives: Bigdata/Hadoop

quorum-based Journal Node으로 hadoop namenode HA 구성하기

by apollo89 Posted on 2013년 1월 25일

Hadoop 의 HA 구성 때문에 Avatar Node 를 검토하던중에, Cloudera 에서 배포된 CDH4.1에 quorum-based journal node에 대해서 알게 되었다. http://touk.pl/blog/en/2012/10/30/hadoop-ha-setup/ https://ccp.cloudera.com/display/CDH4DOC/CDH4+Installation https://ccp.cloudera.com/display/SUPPORT/CDH4+Downloadable+Tarballs https://ccp.cloudera.com/download/attachments/21438336/CDH4_High_Availability_Guide_4.1.pdf?version=3&modificationDate=1349301318000 영문으로 보기가 벅차서 누가 번역하지 않았을까 해서 찾아보니 역시!! http://charsyam.wordpress.com/2012/11/02/%EB%B0%9C-%EB%B2%88%EC%97%AD-quorum-based-journaling-in-cdh4-1/ 그리도 이미 설치해보신 분들도 있다. http://geekdani.wordpress.com/2012/12/22/hadoop-2-0-journal-node-%EC%84%A4%EC%B9%98/ … Continue reading →

Posted in Bigdata/Hadoop | Tagged Avatar Node, ha, Journal Node, namenode, quorum-based, 하둡 | Leave a comment

Javascript Visualization Library

by apollo89 Posted on 2013년 1월 13일

Javascript Visualization Library 참고.. 16 Javascript Libraries for Visualizations : http://datavisualization.ch/tools/13-javascript-libraries-for-visualizations/ Choice Free JavaScript Data Visualization Library : http://kraskniga.blogspot.kr/2012/05/choice-free-javascript-data.html 20 Fresh JavaScript Data Visualization Libraries : http://sixrevisions.com/javascript/20-fresh-javascript-data-visualization-libraries/ Dojo GFX : http://dojotoolkit.org/features/graphics-and-charting gRaphael : http://g.raphaeljs.com/ Google Chart Tools : https://google-developers.appspot.com/chart/interactive/docs/index … Continue reading →

Posted in Bigdata/Hadoop, Javascript/Html | Tagged Javascript/Html, Library, Visualization | Leave a comment

hadoop-장애(Name node is in safe mode.)

by apollo89 Posted on 2013년 1월 4일

회사에 정전이 일어났다. hadoop 개발 장비가 모두 꺼졌다.. 재부팅을 하고 hadoop namenode 를 구동하니 아래와 같은 에러 발생…

2013-01-03 08:30:29,803 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://namenode:9000/data1/hadoop/filesystem/mapreduce/system
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /data1/hadoop/filesystem/mapreduce/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1992)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1972)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:792)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
        at org.apache.hadoop.ipc.Client.call(Client.java:1066)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at $Proxy5.delete(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy5.delete(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:828)
        at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:234)
        at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2410)
        at org.apache.hadoop.mapred.JobTracker.</init><init>(JobTracker.java:2192)
        at org.apache.hadoop.mapred.JobTracker.</init><init>(JobTracker.java:2186)
        at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
        at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2013-01-03 08:30:29,803 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://namenode:9000/data1/hadoop/filesystem/mapreduce/system

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /data1/hadoop/filesystem/mapreduce/system. Name node is in safe mode.

The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1992)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1972)

at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:792)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

at org.apache.hadoop.ipc.Client.call(Client.java:1066)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

at $Proxy5.delete(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)

at $Proxy5.delete(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:828)

at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:234)

at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2410)

at org.apache.hadoop.mapred.JobTracker.</init><init>(JobTracker.java:2192)

at org.apache.hadoop.mapred.JobTracker.</init><init>(JobTracker.java:2186)

at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)

at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)

at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

Hadoop이 정상적인 종료를 하지 않았을 때, 에러가 나는 것으로 보인다. 비정상적인 종료시 hadoop 은 safe 모드로 이동하는데. 종료시 아래와 같은 … Continue reading →

Posted in Bigdata/Hadoop | Tagged chmod, namenode, safemode, 하둡 | Leave a comment

하둡 파일 시스템과 MR 아키텍처

by apollo89 Posted on 2012년 12월 1일

참고 : http://socurites.com/129 원문 : http://www.hadooper.blogspot.kr/2010/11/platform-base-hdfs-mr.html

Posted in Bigdata/Hadoop, Scrap | Tagged map/reduce, 아키텍처, 하둡 | Leave a comment

apache hcatalog 설치와 삽질기..

by apollo89 Posted on 2012년 8월 10일

apache hcatalog 설치 홈페이지 : http://incubator.apache.org/hcatalog/ 설치환경 : Ubuntu 64bit hadoop-1.0.0, hive-0.8.1, mysql 5.1.63, zookeeper-3.3.4, hcatalog-0.4.0, hbase-0.92.1 hcatalog 설치

$ wget http://mirror.apache-kr.org/incubator/hcatalog/hcatalog-0.4.0-incubating/hcatalog-src-0.4.0-incubating.tar.gz
$ tar xvzf hcatalog-src-0.4.0-incubating.tar.gz
$ cd hcatalog-src-0.4.0-incubating
$ ant -Dhcatalog.version=0.4.0 tar
$ cp /home/hcatalog/hcatalog-src-0.4.0-incubating/build/hcatalog-0.4.0.tar.gz /home/hcatalog/
$ cd /home/hcatalog/
$ tar xvzf hcatalog-0.4.0.tar.gz
$ ln -s hcatalog-0.4.0/ hcatalog

$ wget http://mirror.apache-kr.org/incubator/hcatalog/hcatalog-0.4.0-incubating/hcatalog-src-0.4.0-incubating.tar.gz

$ tar xvzf hcatalog-src-0.4.0-incubating.tar.gz

$ cd hcatalog-src-0.4.0-incubating

$ ant -Dhcatalog.version=0.4.0 tar

$ cp /home/hcatalog/hcatalog-src-0.4.0-incubating/build/hcatalog-0.4.0.tar.gz /home/hcatalog/

$ cd /home/hcatalog/

$ tar xvzf hcatalog-0.4.0.tar.gz

$ ln -s hcatalog-0.4.0/ hcatalog

hive는 이미 설치되어 있으므로 생략 한다. hcatalog를 서버로 구동하기 위한 스크립트 실행했더니 아래와 같은 사용법이 나왔다.

$ ./share/hcatalog/scripts/hcat_server_install.sh
Usage: ./share/hcatalog/scripts/hcat_server_install.sh -r root -d dbroot -h hadoop_home -p server_port
root is the directory where you like to install the HCatalog server
/usr/local/hcat is suggested.
dbroot is the directory where your mysql connector jar is located.
hadoop_home is the directory of your Hadoop installation.
server_port is the listening port of the HCatalog server
All paths must be absolute

$ ./share/hcatalog/scripts/hcat_server_install.sh

Usage: ./share/hcatalog/scripts/hcat_server_install.sh -r root -d dbroot -h hadoop_home -p server_port

root is the directory where you like to install the HCatalog server

/usr/local/hcat is suggested.

dbroot is the directory where your mysql connector jar is located.

hadoop_home is the directory of your Hadoop installation.

server_port is the listening port of the HCatalog server

All paths must be absolute

입력 파라미터는 … Continue reading →

Posted in Bigdata/Hadoop | Tagged Hcatalog, Hive, 설치 | 3 Comments

Apache Hadoop Ecosystem 소개

by apollo89 Posted on 2012년 7월 18일

Apache Hadoop Ecosystem은 HDFS를 비롯하여 여러 오픈소스 프로젝트로 구성되어 있다. 대표적인 프로젝트는 다음과 같다. <Big Data Analysis> * HDFS: Hadoop Distributed Filesystem (http://wiki.apache.org/hadoop/HDFS) <- 사용중 * Fuse-DFS: Mountable HDFS (http://wiki.apache.org/hadoop/MountableHDFS/) * HBase: NoSQL (http://hbase.apache.org/) <- 사용중 * HCatalog: 테이블 관리 … Continue reading →

Posted in Bigdata/Hadoop | Tagged Ecosystem, 빅데이터, 하둡 | Leave a comment

R 스터디 내용 정리

by apollo89 Posted on 2012년 7월 12일

R 스터디 내용 정리 1. R 설치 – 홈페이지 둘러보기 (http://www.r-project.org/) – CRAN -> Korean -> Download R for MacOS X -> 설치 2. Rstudio 설치 – 홈페이지 둘러보기 (http://rstudio.org/) – Download RStudio -> 설치 3. 구동 및 실행 … Continue reading →

Posted in Bigdata/Hadoop | Tagged KRUG, R, r-project, Rstudio, 설치 | Leave a comment

주목! 9가지 오픈소스 빅 데이터 기술

by apollo89 Posted on 2012년 7월 1일

주목! 9가지 오픈소스 빅 데이터 기술 갈수록 많은 기업들이 더 많은 데이터를 축적하며 경쟁력 향상을 꾀하고 있다. 그리고 이와 같은 빅 데이터 열풍의 중심에는 오픈소스 기술이 자리 잡고 있다. 여기 독자 여러분이 주목해야 할 아홉 개의 오픈소스 빅 데이터 … Continue reading →

Posted in Bigdata/Hadoop, Scrap | Tagged Cascading, Cassandra, CouchDB, ElasticSearch, HBase, MongoDB, R, Scribe, 빅데이터, 오픈소스, 하둡 | Leave a comment

Flume 설치

by apollo89 Posted on 2012년 4월 21일

1. Flume 설치

]$ wget https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz
]$ tar xvzf flume-distribution-0.9.4-bin.tar.gz
]$ ln -s flume-distribution-0.9.4 flume
]$ cd /service/workspace/flume

]$ wget https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz

]$ tar xvzf flume-distribution-0.9.4-bin.tar.gz

]$ ln -s flume-distribution-0.9.4 flume

]$ cd /service/workspace/flume

2 Flume 설정 2.1 flume-site.xml 설정

]$ vi conf/flume-site.xml
...

    flume.master.servers
    localhost
    This is the address for the config servers status server (http)

...

]$ vi conf/flume-site.xml

...

flume.master.servers

localhost

This is the address for the config servers status server (http)

...

2.2 flume-env.sh 설정

]$ vi bin/flume-env.sh
...
export FLUME_HOME=/service/workspace/flume
export FLUME_PID_DIR=/service/workspace/flume/pid
export FLUME_LOG_DIR=/service/workspace/flume/logs
...

]$ vi bin/flume-env.sh

...

export FLUME_HOME=/service/workspace/flume

export FLUME_PID_DIR=/service/workspace/flume/pid

export FLUME_LOG_DIR=/service/workspace/flume/logs

...

2.3 logs 디렉토리 생성

]$ mkdir logs

1	]$ mkdir logs

3 Flume 구동

]$ ./bin/flume-daemon.sh start node -n agent_name

1	]$ ./bin/flume-daemon.sh start node -n agent_name

4 Flume 확인 http://localhost:35871/flumemaster.jsp * Flume Master 에서 agent_name으로 올라오는 정보가 있는지 확인 설치 끝! … Continue reading →

Posted in Bigdata/Hadoop | Tagged Flume, 설치 | Leave a comment

Hive 설치

by apollo89 Posted on 2012년 4월 21일

1 Hive 설치 설치 위치 : /service/workspace (각자 환경에 맞게 구성)

]$ wget http://mirror.apache-kr.org/hive/hive-0.8.1/hive-0.8.1.tar.gz
]$ tar xvzf hive-0.8.1.tar.gz
]$ ln -s hive-0.8.1 hive
]$ cd /service/workspace/hive&nbsp;

]$ wget http://mirror.apache-kr.org/hive/hive-0.8.1/hive-0.8.1.tar.gz

]$ tar xvzf hive-0.8.1.tar.gz

]$ ln -s hive-0.8.1 hive

]$ cd /service/workspace/hive 

2 Hive 설정 2.1 hive-site.xml 설정

]$ vi conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
          <name>javax.jdo.option.ConnectionURL</name>
          <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value>
        </property>
        <property>
          <name>javax.jdo.option.ConnectionDriverName</name>
          <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
          <name>javax.jdo.option.ConnectionUserName</name>
          <value>dbuser</value>
        </property>
        <property>
          <name>javax.jdo.option.ConnectionPassword</name>
          <value>dbpass</value>
        </property>
</configuration>

]$ vi conf/hive-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>dbuser</value>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>dbpass</value>

</property>

</configuration>

* 먼저 Mysql 접속 계정이 있어야 함(mysql 설치는 별도로) * 원래는 hive-default.xml.template 파일을 복사해서 사용해야 하지만 mysql접속정보만 설정하기 위해서 따로 … Continue reading →

Posted in Bigdata/Hadoop | Tagged Hive, 설치, 하이브 | Leave a comment

Apollo89.com

아폴로씨의 잡다한 경험들..

Category Archives: Bigdata/Hadoop

quorum-based Journal Node으로 hadoop namenode HA 구성하기

Javascript Visualization Library

hadoop-장애(Name node is in safe mode.)

하둡 파일 시스템과 MR 아키텍처

apache hcatalog 설치와 삽질기..

Apache Hadoop Ecosystem 소개

R 스터디 내용 정리

주목! 9가지 오픈소스 빅 데이터 기술

Flume 설치

Hive 설치

카테고리

최신 댓글

메타