MapReduce In Hadoop: Everything You Wanted to Know About

Do you want to know about MapReduce In Hadoop? If yes, then give your few minutes to this blog to know What is MapReduce In Hadoop and all details related to MapReduce In Hadoop.

Hello, & Welcome!

In this blog, I am gonna tell you-

1. What is MapReduce In Hadoop?
2. MapReduce Process.
3. Input Format in MapReduce.
4. How does Word Count Program work in MapReduce?

Firstly, I would like to start with-

What is MapReduce In Hadoop?

MapReduce is a Software framework and programming model, which you can use for processing a large number of data. As the name suggests, MapReduce consists of two parts, the Map and Reduce. Where Map performs splitting and mapping of the data and Reduce performs Shuffling and reducing the data.

You can write a MapReduce program in Java, Python, C++, and Ruby. The MapReduce process a large amount of unstructured data with a distributed algorithm. Therefore MapReduce is the core element for processing the data in Hadoop.

MapReduce Process.

MapReduce has six phases-

1. Mapper Task
2. Combiner Task
3. Partition Task
4. Sorting Task
5. Grouping Task
6. Reducer Task

Where in Mapper task main logic is written and Reducer Task performs aggregates like sum, add, etc. Combiner, Partition, Sorting, and Grouping are done automatically. You don’t need to do anything. Only you have to work on Mapper and Reducer.

Input is given to the Mapper task and you can get an output at Reducer Task after all the processing has been done.

We’ll understand the whole procedure in the next section of the word count program.

Input Format in MapReduce.

In Mapper, you have to provide input in key, value pair something like that <K, V>.

So <K, V> pair has two types of input format-

1. Text Input Format
2. Sequence Input Format

1. Text Input Format-

The Text input format generates the Key automatically. That’s why it is good for BigData, where we don’t have a need to define every key.

Therefore in the text input format,

k -> is system generated number

V-> is the remaining data( means the data which we provide as an input to Mapper)

2. Sequence Input Format-

In the sequence input format, the user has to define the key. It is good only in small size files. If the file size is long, it is not good because the user has to define all the keys in a long file. Which is very time taking process.

Therefore in the sequence input format,

K -> is user-defined.

V-> is the remaining data( means the data which we provide as an input to Mapper)

How does Word Count Program work in MapReduce?

Suppose you have to count the word in a big file. So how can you perform this task using MapReduce? Here, I am gonna tell you the whole procedure of how MapReduce counts the word, and what different phases of MapReduce perform their job.

As I have discussed to you that MapReduce has six phases, here we will understand each phase functionality in details.

So, Let’s start-

Suppose this is the file, where we have to count the words-

Words

B

A

C

B

A

C

B

A

A

Here, we have to count the count of A’s, count of B’s, and count of C’s.

The first phase in MapReduce is-

1. Mapper Task-

Here, I am gonna use the Text input format, so the key is generated automatically. We don’t need to define manually.

So, after applying text input format we get a list something like that-

Mapper Input-

K	V
–	B
–	A
–	C
–	B
–	A
–	C
–	B
–	A
–	A

Here, keys may be auto-generated, that’ why I put -, because it may be anything. This is HDFS Mapper Input. Here Mapper code applies and when we run mapper code, we get Mapper Output, which looks something like that-

Mapper Output-

K	V
B	1
A	1
C	1
B	1
A	1
C	1
B	1
A	1
A	1

2. Combiner Task-

The combiner task is done automatically, you don’t need to perform anything. The Combiner task simply combines all A’s, B’s, and C’s together.

3. Partition Task-

This task happened automatically. It simply partitions each words something like that-

K	V
B	1
B	1
B	1
A	1
A	1
A	1
A	1
C	1
C	1

Here, it partitions A, B, and C.

4. Sorting Task-

Sorting also happened automatically. Here, it sorts in the ascending order, which means all A’s come first, then B’s and then all C’s. After Partition task, we get file something like that-

K	V
A	1
A	1
A	1
A	1
B	1
B	1
B	1
C	1
C	1

I hope, you are understanding MapReduce process till now 🙂

5. Grouping Task-

Grouping also happened automatically. It group all A’s, B’s, and C’s values together. In simple words, it group all the words value together. After Grouping Task, the data looks something like that-

K	g(V)
A	(1,1,1,1)
B	(1,1,1)
C	(1,1)

This is the Input for the Reducer Task.

6. Reducer Task-

Here, you need to perform operations like sum, add or anything else depending upon you. So the Grouping Task data act as Reducer Input. That means you have to pass Grouping data as a Reducer Input.

Let’s see-

Reducer Input-

K	g(V)
A	(1,1,1,1)
B	(1,1,1)
C	(1,1)

On that input data, you need to perform operations in order to get the final output. In that example, we have to count the words. Therefore after applying some operations, we get the final output as-

K	s(V)
A	4
B	3
C	2

Hurray! We get our final output as word count.

I hope, now you understand MapReduce’s full functionality.

Congratulations! That’s all for MapReduce in Hadoop.

Enjoy Learning!

All the Best!

Related Search

Best Online Courses for Data Science to become A Skilled Data Scientist

15 Best Books on Data Science Everyone Should Read in 2024
Data Science vs Data Analyst: Ultimate Guide to Clear Doubts
How to make Data Science Resume to Get Hired?
What is Big Data Analytics? Things no one tells you
Data Science: Top 8 Most Demanding Skills to Get You Hired

Explore More about Data Science, Visit Here

Thank YOU!

Though of the Day…

‘ It’s what you learn after you know it all that counts.’

– John Wooden