apache hcatalog 설치
홈페이지 : http://incubator.apache.org/hcatalog/
설치환경 : Ubuntu 64bit
hadoop-1.0.0, hive-0.8.1, mysql 5.1.63, zookeeper-3.3.4, hcatalog-0.4.0, hbase-0.92.1
hcatalog 설치
|
1 2 3 4 5 6 7 8 |
$ wget http://mirror.apache-kr.org/incubator/hcatalog/hcatalog-0.4.0-incubating/hcatalog-src-0.4.0-incubating.tar.gz $ tar xvzf hcatalog-src-0.4.0-incubating.tar.gz $ cd hcatalog-src-0.4.0-incubating $ ant -Dhcatalog.version=0.4.0 tar $ cp /home/hcatalog/hcatalog-src-0.4.0-incubating/build/hcatalog-0.4.0.tar.gz /home/hcatalog/ $ cd /home/hcatalog/ $ tar xvzf hcatalog-0.4.0.tar.gz $ ln -s hcatalog-0.4.0/ hcatalog |
hive는 이미 설치되어 있으므로 생략 한다.
hcatalog를 서버로 구동하기 위한 스크립트 실행했더니 아래와 같은 사용법이 나왔다.
|
1 2 3 4 5 6 7 8 |
$ ./share/hcatalog/scripts/hcat_server_install.sh Usage: ./share/hcatalog/scripts/hcat_server_install.sh -r root -d dbroot -h hadoop_home -p server_port root is the directory where you like to install the HCatalog server /usr/local/hcat is suggested. dbroot is the directory where your mysql connector jar is located. hadoop_home is the directory of your Hadoop installation. server_port is the listening port of the HCatalog server All paths must be absolute |
입력 파라미터는 아래와 같다.
dbroot : mysql connector jar 가 있는 위치
hadoop_home : Hadoop 이 설치된 위치
server_port : HCatalog server가 사용할 port
아래의 값을 입력하고 다시 구동..
|
1 2 |
$ ./share/hcatalog/scripts/hcat_server_install.sh -r /home/hcatalog/hcatalog/ -d /home/hcatalog/hcatalog/lib -h /home/hadoop/hadoop -p 10001 Your root directory must be readable, writable, and executable by its owner. |
root 디렉토리에 rwx권한이 필요하다고 합니다.
|
1 |
$ sudo chmod 777 /home/hcatalog/hcatalog-0.4.0/ |
다시 시도!
|
1 2 3 |
$ ./share/hcatalog/scripts/hcat_server_install.sh -r /home/hcatalog/hcatalog/ -d /home/hcatalog/hcatalog/lib -h /home/hadoop/hadoop -p 10001 Installing into [/home/hcatalog/hcatalog/] Installation successful |
환경변수 설정
|
1 |
$ export HCAT_HOME=/home/hcatalog/hcatalog |
hive설정 파일 세팅
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
$ sudo vi /home/hive/hive/conf/hive-site.xml < ?xml version="1.0"?> < ?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration> <property> <name>hive.metastore.execute.setugi</name> <value>true</value> <description>In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored.</description> </property> <property> <name>hive.metastore.local</name> <value>false</value> <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>user</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>password</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>WAREHOUSE_DIR</value> <description>location of default database for the warehouse</description> </property> <property> <name>hive.metastore.sasl.enabled</name> <value>SASL_ENABLED</value> <description>If true, the metastore thrift interface will be secured with SASL. Clients must authenticate with Kerberos.</description> </property> <property> <name>hive.metastore.kerberos.keytab.file</name> <value>KEYTAB_PATH</value> <description>The path to the Kerberos Keytab file containing the metastore thrift server's service principal.</description> </property> <property> <name>hive.metastore.kerberos.principal</name> <value>KERBEROS_PRINCIPAL</value> <description>The service principal for the metastore thrift server. The special string _HOST will be replaced automatically with the correct host name.</description> </property> <property> <name>hive.metastore.cache.pinobjtypes</name> <value>Table,Database,Type,FieldSchema,Order</value> <description>List of comma separated metastore object types that should be pinned in the cache</description> </property> <property> <name>hive.metastore.uris</name> <value>thrift://127.0.0.1:10001</value> <description>URI for client to contact metastore server</description> </property> <property> <name>hive.semantic.analyzer.factory.impl</name> <value>org.apache.hcatalog.cli.HCatSemanticAnalyzerFactory</value> <description>controls which SemanticAnalyzerFactory implemenation class is used by CLI</description> </property> <property> <name>hadoop.clientside.fs.operations</name> <value>true</value> <description>FS operations are owned by client</description> </property> <property> <name>hive.metastore.client.socket.timeout</name> <value>60</value> <description>MetaStore Client socket timeout in seconds</description> </property> <property> <name>hive.security.authorization.enabled</name> <value>true</value> <description>enable or disable the hive client authorization</description> </property> <property> <name>hive.security.authorization.manager</name> <value>org.apache.hcatalog.security.StorageDelegationAuthorizationProvider</value> <description>the hive client authorization manager class name. The user defined authorization class should implement interface org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider. HCatalog uses a model, where authorization checks are delegated to the storage layer (hdfs, hbase, ...). </description> </property> </configuration> |
편리한 사용을 위해 hive의 설정 파일과 실행파일을 hcatalog에 심볼릭링크.
|
1 2 |
$ sudo ln -s /home/hive/hive/conf/hive-site.xml /home/hcatalog/hcatalog/etc/hcatalog/hive-site.xml $ sudo ln -s /home/hive/hive/bin/hive /home/hcatalog/hcatalog/bin/hive |
hcatalog 서버 구동!
|
1 2 |
$ ./sbin/hcat_server.sh start Hadoop not found. |
흠.. Hadoop이 없데서 bashrc에 hadoop과 hive path추가
|
1 2 3 4 |
$ vi /home/hcatalog/.bashrc ... export HIVE_HOME=/home/hive/hive export HADOOP_HOME=/home/hadoop/hadoop |
다시 구동!
|
1 2 3 |
$ ./sbin/hcat_server.sh start Started metastore server init, testing if initialized correctly... $ ./sbin/hcat_server.sh: 줄 89: /home/hcatalog/hcatalog/sbin/../var/log/hcat.out: 그런 파일이나 디렉터리가 없습니다 |
흠냐.. 이번엔 로그파일을 못만든다네.. 디렉토리 생성
|
1 2 |
$ mkdir /home/hcatalog/hcatalog/var $ mkdir /home/hcatalog/hcatalog/var/log |
다시 구동!
|
1 2 3 |
$ ./sbin/hcat_server.sh start Started metastore server init, testing if initialized correctly... Metastore initialized successfully on port[10001]. |
설치 성공! 우후훗! (도움을 주신 GeekDani 님 감사 ^^)
참고 : http://mixellaneous.tistory.com/1123
3 Responses to apache hcatalog 설치와 삽질기..