Installing Sun-Java JDK Using Bash Script :: Ubuntu

Getting Started

We are going to write simple bash script and execute it to install Sun Java. We will write content to text file using any of your favorite test editor. give it permission to execute and enjoy. hope it  works without error, else let me know issue to update it, it works fine for me on 2 different machines running Linux mint 17 and Ubuntu 14.04 respectively.

What is Bash?

Descended from the Bourne Shell, Bash is a GNU product, the "Bourne Again SHell." It's the standard command line interface on most Linux machines. It excels at interactivity, supporting command line editing, completion, and recall. It also supports configurable prompts - most people realize this, but don't know how much can be done.

Writing Script:

To write script for installation open new file named install_java.sh and put following content in it.

#!/bin/bash

# Script to install Sun Java

clear

# This command is used to tell shell, turn installation mode to non interactive
# and set auto selection of agreement for Sun Java
export DEBIAN_FRONTEND=noninteractive
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections


echo "Bash Script for Installing Sun Java for Ubuntu!"

echo "Now Script will try to purge OpenJdk if installed..."

# purge openjdk if installed to remove conflict
sudo apt-get purge openjdk-\* -y

echo "Now we will update repository..."

sudo apt-get update -y

echo "Adding Java Repository...."

sudo apt-get install python-software-properties -y
sudo add-apt-repository ppa:webupd8team/java -y

echo "Updating Repository to load java repository"

sudo apt-get update -y

echo "Installing Sun Java....."
sudo -E apt-get purge oracle-java7-installer -y
sudo -E apt-get install oracle-java7-installer -y

echo "Installation completed...."

echo "Installed java version is...."

java -version
   

Now we will give it permission to make it executable.

 ~$ chmod 755 install_java.sh

Now we can execute script using following command, Since installation require root excess and use sudo, script may prompt to ask password. provide password to initiate installation process.

 ~$ ./install_java.sh  

Enjoy.

Apache Hive? Compile Hive with Ubuntu 14 :: HADOOP 2.6.0

Get Started Hive

The Apache Hive ™ is data warehouse infrastructure built on top of Hadoop that software facilitates querying and managing large data-sets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL. For details visit WIKI

By default, Hive stores metadata in an embedded Apache Derby database, and other client/server databases like MySQL, PostgreSQL can optionally be used. While Yahoo was working with PIG for deployment on Hadoop, Facebook started their own warehouse solutions on Hadoop which resulted on HIVE, The reason behind using hive is because traditional warehousing  solutions are getting expensive, While initially developed by Facebook, Apache Hive is now used and developed by other companies such as Netflix.

Components of Hive:

HCatalog is a component of Hive. It is a table and storage management layer for Hadoop that enables users with different data processing tools — including Pig and MapReduce — to more easily read and write data on the grid.

WebHCat provides a service that you can use to run Hadoop MapReduce (or YARN), Pig, Hive jobs or perform Hive metadata operations using an HTTP (REST style) interface.

Hive is not built for quick response applications thus, can not be compared with other applications designed for reduced response times. it is build for data mining applications with post processing of data distributed over Hadoop cluster.

features of Hive include:

• Indexing to provide acceleration
• Different storage types such as plain text, RCFile, HBase, ORC, and others.
• Metadata storage in an RDBMS, significantly reducing the time to perform semantic checks during query execution.
• Operating on compressed data stored into the Hadoop ecosystem using algorithms including DEFLATE, BWT, snappy, etc.
• Built-in user defined functions (UDFs) to manipulate dates, strings, and other data-mining tools. Hive supports extending the UDF set to handle use-cases not supported by built-in functions.
• SQL-like queries (HiveQL), which are implicitly converted into Map-reduce, or Spark jobs.

HiveQL

HiveQL is based on SQL, but it does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multi-table inserts and create table as select, but only offers basic support for indexes. Also, HiveQL lacks support for transactions and materialized views, and only limited sub-query support. Support for insert, update, and delete with full ACID functionality was made available with release 0.14.

Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce or Tez, or Spark jobs, which are submitted to Hadoop for execution.

Data is hive is organized in three formats.

Tables: They are very similar to RDBMS tables and contains rows and tables. Hive is just layered over the Hadoop File System (HDFS), hence tables are directly mapped to directories of the file-systems. It also supports tables stored in other native file systems.

Partitions: Hive tables can have more than one partition. They are mapped to sub-directories and file systems as well.

Buckets: In Hive data may be divided into buckets. Buckets are stored as files in partition in the underlying file system.

Hive store metastore in relational database containing metadata about hive schema.

For details on how to use HiveQL check following Wikis.

DDL Operations

DML Operations

SQL Operations

Prerequisite for Hive

since hive is based on Hadoop and uses HDFS, Hadoop is required to be installed prior to Hive installation, we have used Hadoop 2.6.0 latest release till date. to install hadoop is not configured earlier, follow my post on How to Install Hadoop 2.6.0

Next we require Apache Maven and Subversion. Apache Maven is required to build Apache Hive while subversion is required to clone source for compilation. to resolve Apache Maven version conflict issue we will remove maven2 before maven installation if installed. 

 ~$ sudo apt-get update  
 ~$ sudo apt-get remove maven2  
 ~$ sudo apt-get install maven  
   
 ~$ apt-get install subversion  
   

Compile Hive

after prerequisite for hive installed, we can move forward to compile Hive, We will clone source from repository using subversion and will build it using maven.

 svn co http://svn.apache.org/repos/asf/hive/trunk hive  
 cd hive  
   
 mvn clean install -Phadoop-2,dist -e -DskipTests  

Now compilation is completed successfully, we will export Hive path, change version number from export command if changed, this can be verified by listing directory.

 ~$ nano ~/.bashrc  

Add following line:

 export HIVE_HOME=/root/hive/packaging/target/apache-hive-1.2.0-SNAPSHOT-bin/apache-hive-1.2.0-SNAPSHOT-bin  

Save and close file and reload bachrc using source command.

 ~$ source ~/.bashrc  

Now we will create warehouse for Hive in Hadoop, temp directory and set permissions for write.

 ~$ $HADOOP_HOME/bin/hadoop fs -mkdir    /tmp  
 ~$ $HADOOP_HOME/bin/hadoop fs -mkdir    /user/  
 ~$ $HADOOP_HOME/bin/hadoop fs -mkdir    /user/hive/  
 ~$ $HADOOP_HOME/bin/hadoop fs -mkdir    /user/hive/warehouse  
   
 ~$ $HADOOP_HOME/bin/hadoop fs -chmod g+w  /tmp  
 ~$ $HADOOP_HOME/bin/hadoop fs -chmod g+w  /user/hive/warehouse  

To start Beeline CLI use following.

 ~$ $HIVE_HOME/bin/beeline -u jdbc:hive2://  

Now we are ready to use Hive, for testing we have practiced Hive and provided is snap of shell. We have practiced this with 8GB RAM with 4 cores of CPU on SSD Drives.

 0: jdbc:hive2://> create database mydb;  
 15/04/22 08:31:23 [HiveServer2-Background-Pool: Thread-32]: WARN metastore.ObjectStore: Failed to get database mydb, returning NoSuchObjectException  
 OK  
 No rows affected (1.565 seconds)  
 0: jdbc:hive2://> CREATE TABLE mydb.testdata (  
 0: jdbc:hive2://>   id  INT,  
 0: jdbc:hive2://>   data VARCHAR(30)  
 0: jdbc:hive2://> );  
 OK  
 No rows affected (0.561 seconds)  
 0: jdbc:hive2://> INSERT into mydb.testdata(id, data) values(1, 'Testing 1');  
 Query ID = root_20150422083821_5e2e9f2c-dc47-4650-9280-a9e52cb61c7c  
 Total jobs = 3  
 Launching Job 1 out of 3  
 Number of reduce tasks is set to 0 since there's no reduce operator  
 15/04/22 08:38:22 [HiveServer2-Background-Pool: Thread-59]: ERROR mr.ExecDriver: yarn  
 15/04/22 08:38:23 [HiveServer2-Background-Pool: Thread-59]: WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.  
 Starting Job = job_1429700414970_0001, Tracking URL = http://hadoop:8088/proxy/application_1429700414970_0001/  
 Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1429700414970_0001  
 WARN : Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.  
 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0  
 15/04/22 08:38:31 [HiveServer2-Background-Pool: Thread-59]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead  
 2015-04-22 08:38:31,952 Stage-1 map = 0%, reduce = 0%  
 2015-04-22 08:38:39,433 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.24 sec  
 MapReduce Total cumulative CPU time: 2 seconds 240 msec  
 Ended Job = job_1429700414970_0001  
 Stage-4 is selected by condition resolver.  
 Stage-3 is filtered out by condition resolver.  
 Stage-5 is filtered out by condition resolver.  
 Moving data to: hdfs://localhost:9000/user/hive/warehouse/mydb.db/testdata/.hive-staging_hive_2015-04-22_08-38-21_684_4213030447536309872-1/-ext-10000  
 Loading data to table mydb.testdata  
 Table mydb.testdata stats: [numFiles=1, numRows=1, totalSize=12, rawDataSize=11]  
 MapReduce Jobs Launched:  
 Stage-Stage-1: Map: 1  Cumulative CPU: 2.24 sec  HDFS Read: 3732 HDFS Write: 77 SUCCESS  
 Total MapReduce CPU Time Spent: 2 seconds 240 msec  
 OK  
 No rows affected (19.328 seconds)  
 0: jdbc:hive2://> SELECT * FROM mydb.testdata;  
 OK  
 +----------+------------+--+  
 | testdata.id | testdata.data |  
 +----------+------------+--+  
 | 1    | Testing 1 |  
 +----------+------------+--+  
 1 row selected (0.223 seconds)  
   
 0: jdbc:hive2://> SELECT * FROM mydb.testdata WHERE testdata.id%2==1;  
 OK  
 +----------+------------+--+  
 | testdata.id | testdata.data |  
 +----------+------------+--+  
 | 1    | Testing 1 |  
 +----------+------------+--+  
 1 row selected (0.129 seconds)  
 0: jdbc:hive2://>  
   

Enjoy.

Installing Hadoop Single Node - 1.2.1

Get Started

Now we will check how to install stable version of Apache Hadoop on a Laptop running Linux Mint 15 but will work on all Debian based systems including Ubuntu. To start we need to acquire hadoop package and get java installed, to install java, if not already installed follow my install java post. to check which versions of java are supported with hadoop check Hadoop Java Versions. Next step is to acquire hadoop which could be downloaded @ hadoop webpage. we opted for hadoop-1.2.1 in our blog.

Create Dedicated Hadoop User

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hdpuser

Give user sudo rights

$ sudo nano /etc/sudoers
add this to end of file
hdpuser ALL=(ALL:ALL) ALL

Configuring Secure Shell (SSH)   

Communication between master and slave nodes uses SSH, to ensure we have SSH server installed
and running SSH deamon.

Installed server with provided command:

$ sudo apt-get install openssh-server

You can check status of server use this command

$ /etc/init.d/ssh status

To start ssh server use:

$ /etc/init.d/ssh start

Now ssh server is running, we need to set local ssh connection with password. To enable passphraseless ssh use

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

to check ssh

$ ssh localhost
$ exit

Disabling IPv6

We need to make sure IPv6 is disabled, it is best to disable IPv6 as all Hadoop communication between nodes is IPv4-based.

For this, first access the file /etc/sysctl.conf

$ sudo nano /etc/sysctl.conf
add following lines to end
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Save and exit

Reload sysctl for changes to take effect

$ sudo sysctl -p /etc/sysctl.conf

If the following command returns 1 (after reboot), it means IPv6 is disabled.

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

Install Hadoop

Download Version 1.2.1 (Stable Version)

Make Hadoop installation directory

$ sudo mkdir -p /usr/hadoop

Copy Hadoop installer to installation directory

$ sudo cp -r ~/Downloads/hadoop-1.2.1.tar.gz /usr/hadoop

Extract Hadoop installer

$ cd /usr/hadoop
$ sudo tar xvzf hadoop-1.2.1.tar.gz

Rename it to hadoop

$ sudo mv hadoop-1.2.1 hadoop

Change owner to hdpuser for this folder

$ sudo chown -R hdpuser:hadoop hadoop

Update .bashrc with Hadoop-related environment variables

$ sudo nano ~/.bashrc
Add following lines at the end:
# Set HADOOP_HOME
export HADOOP_HOME=/usr/hadoop/hadoop
# Set JAVA_HOME
# Import if you have installed java from apt-get
# use /usr instead of /usr/local/java/jdk1.7.0_51
export JAVA_HOME=/usr/local/java/jdk1.7.0_51
# Add Hadoop bin directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin

Save & Exit

Reload bashrc

$ source ~/.bashrc

Update JAVA_HOME in hadoop-env.sh

$ cd /usr/hadoop/hadoop
$ sudo nano conf/hadoop-env.sh

Add the line:
export JAVA_HOME=/usr/local/java/jdk1.7.0_51

Save and exit

Create a Directory to hold Hadoop’s Temporary Files:

$ sudo mkdir -p /usr/hadoop/tmp

Provide hdpuser the rights to this directory

$ sudo chown hdpuser:hadoop /usr/hadoop/tmp

Hadoop Configurations

Modify conf/core-site.xml – Core Configuration

$ sudo nano conf/core-site.xml

Add the following lines between configuration tags
<property>
   <name>hadoop.tmp.dir</name>
   <value>/usr/hadoop/tmp</value>
   <description>Hadoop's temporary directory</description>
</property>
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:54310</value>
   <description>Specifying HDFS as the default file system.</description>
</property>

Modify conf/mapred-site.xml – MapReduce configuration

$ sudo nano conf/mapred-site.xml

Add the following lines between configuration tags
<property>
   <name>mapred.job.tracker</name>
   <value>localhost:54311</value>
   <description>The URI is used to monitor the status of MapReduce tasks</description>
</property>

Modify conf/hdfs-site.xml – File Replication

$ sudo nano conf/hdfs-site.xml

Add following lines between configuration tags:
<property>
   <name>dfs.replication</name>
   <value>1</value>
   <description>Default block replication.</description>
</property>

Initializing the Single-Node Cluster

Formatting the Name Node:

While setting up the cluster for the first time, we need to initially format the Name Node in HDFS.
$ bin/hadoop namenode -format

Starting all daemons:

$ bin/start-all.sh

You should now be able to browse the nameNode and JobTracker in your browser (after a short delay for startup) by browsing to the following URLs:

nameNode: http://localhost:50070/
JobTracker: http://localhost:50030/

Stoping all daemons:

$ bin/stop-all.sh

your can seperatly start stop as

hdfs:

$ bin/start-dfs.sh
$ bin/stop-dfs.sh

mappered:

$ bin/start-mapred.sh
$ bin/stop-mapred.sh


Now run examples Java Word Count Example.  looking for examples to run without changing your style of code, am going run python map-reduce wait for post.

Installing Sun-Java JDK 7

Install Java JDK OS- MINT15, will work on on all Debian Based systems including ubuntu Easy Way: Simple and easy way to install JDK is to do it with apt-get repository. but noted that this some ti PPA becomes out dated.  This installs JDK 7 (which includes Java JDK, JRE and the Java browser plugin). Remove any installed version of open-JDK $ sudo apt-get purge openjdk-\* Add PPA and update apt-get repo $ sudo apt-get update $ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update install it $ sudo apt-get install oracle-java7-installer Check version and process id Check your Java version to ensure installations and settings: $ java -version Verify that JPS (JVM Process Status tool) is up and running $ jps Manual way: 1) Remove any previous OpenJDK installations $ sudo apt-get purge openjdk-\* 2) Make directory to hold Sun Java $ sudo mkdir -p /usr/local/java 3) Download Oracle Java Sun (JDK/JRE) from Oracle’s website: JDK Download and JRE Download. Normally downloaded files will be placed in /home/<your_user_name>/Downloads folder. 4) Copy the downloaded files to the Java directory $ cd /home/<your_user_name>/Downloads $ sudo cp -r jdk-7u51-linux-x64.tar.gz /usr/local/java $ sudo cp -r jre-7u51-linux-x64.tar.gz /usr/local/java 5) Unpack the compressed binaries $ cd /usr/local/java $ sudo tar xvzf jdk-7u51-linux-x64.tar.gz $ sudo tar xvzf jre-7u51-linux-x64.tar.gz 6) Cross-check the extracted binaries: $ ls -a The following two folders should be created: jdk1.7.0_51 and jre1.7.0_51 7) To provide information about JDK/JRE paths to the system PATH (located in /etc/profile), first access the PATH: $sudo nano /etc/profile and add the following lines at the end: JAVA_HOME=/usr/local/java/jdk1.7.0_51 PATH=$PATH:$HOME/bin:$JAVA_HOME/bin JRE_HOME=/usr/local/java/jre1.7.0_51 PATH=$PATH:$HOME/bin:$JRE_HOME/bin export JAVA_HOME export JRE_HOME export PATH Save and exit (CTRL+O then Enter, then press CTRL+X then Enter) 8) Inform OS about Oracle Sun Java location to signal that it is ready for use: JDK is available: $ sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_51/bin/javac" 1 JRE is available: $ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jre1.7.0_51/bin/java" 1 Java Web Start is available: $ sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jre1.7.0_51/bin/javaws" 1 9) Make Oracle Sun JDK/JRE the default on your system: Set JRE: $ sudo update-alternatives --set java /usr/local/java/jre1.7.0_51/bin/java Set javac Compiler: $ sudo update-alternatives --set javac /usr/local/java/jdk1.7.0_51/bin/javac Set Java Web Start: $ sudo update-alternatives --set javaws /usr/local/java/jre1.7.0_51/bin/javaws 10) Re-load the /etc/profile $ source /etc/profile 11) Check your Java version to ensure installations and settings: $ java -version 12) Verify that JPS (JVM Process Status tool) is up and running $ jps This will show the process id of the jps process