Hadoop Mapreduce Example - WordCount.java
공부/하둡(Hadoop) / 2010. 5. 18. 17:47
Other people do this without any problems. However why I can't do this!!
That because there is some missing line at blogs and web pages which I found from google. Holy shit.
You can run with this command very nicely. Your welcome? No Sweat.
$ mkdir wordcount_classes
$ javac -classpath hadoop-0.20.2-core.jar -d wordcount_classes WordCount.java
$ jar cvf wordcount.jar -C wordcount_classes/ .
$ mkdir -p wordcount/input
$ echo "Hello Word Bye World" > wordcount/input/file01
$ echo "Hello Hadoop Goodbye Hadoop" > wordcount/input/file02
$ bin/hadoop fs -put wordcount/input wordcount/input
$ bin/hadoop dfs -ls wordcount/input/
Found 2 items
/usr/local/oss/hadoop-0.17.0/wordcount/input/file02 <r 1> 28 2008-08-05 01:44 rw-rw-r-- makoto makoto
/usr/local/oss/hadoop-0.17.0/wordcount/input/file01 <r 1> 21 2008-08-05 01:45 rw-rw-r-- makoto makoto
$ bin/hadoop dfs -cat wordcount/input/file01
Hello World By World
$ bin/hadoop dfs -cat wordcount/input/file02
Hello Hadoop GoodBye Hadoop
$ bin/hadoop jar wordcount.jar org.myorg.WordCount wordcount/input wordcount/output
$ bin/hadoop fs -ls wordcount/output
Found 2 items
drwxr-xr-x - root supergroup 0 2010-01-05 17:46 /user/root/wordcount/output/_logs
-rw-r--r-- 2 root supergroup 41 2010-01-05 17:46 /user/root/wordcount/output/part-00000
$ bin/hadoop fs -cat wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
Reference (Examples from below sites are not perfect)
http://gmyoul.tistory.com/11
http://code.google.com/p/newtech/wiki/HadoopMapReduceFirstStep
That because there is some missing line at blogs and web pages which I found from google. Holy shit.
You can run with this command very nicely. Your welcome? No Sweat.
$ mkdir wordcount_classes
$ javac -classpath hadoop-0.20.2-core.jar -d wordcount_classes WordCount.java
$ jar cvf wordcount.jar -C wordcount_classes/ .
$ mkdir -p wordcount/input
$ echo "Hello Word Bye World" > wordcount/input/file01
$ echo "Hello Hadoop Goodbye Hadoop" > wordcount/input/file02
$ bin/hadoop fs -put wordcount/input wordcount/input
$ bin/hadoop dfs -ls wordcount/input/
Found 2 items
/usr/local/oss/hadoop-0.17.0/wordcount/input/file02 <r 1> 28 2008-08-05 01:44 rw-rw-r-- makoto makoto
/usr/local/oss/hadoop-0.17.0/wordcount/input/file01 <r 1> 21 2008-08-05 01:45 rw-rw-r-- makoto makoto
$ bin/hadoop dfs -cat wordcount/input/file01
Hello World By World
$ bin/hadoop dfs -cat wordcount/input/file02
Hello Hadoop GoodBye Hadoop
$ bin/hadoop jar wordcount.jar org.myorg.WordCount wordcount/input wordcount/output
$ bin/hadoop fs -ls wordcount/output
Found 2 items
drwxr-xr-x - root supergroup 0 2010-01-05 17:46 /user/root/wordcount/output/_logs
-rw-r--r-- 2 root supergroup 41 2010-01-05 17:46 /user/root/wordcount/output/part-00000
$ bin/hadoop fs -cat wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
Reference (Examples from below sites are not perfect)
http://gmyoul.tistory.com/11
http://code.google.com/p/newtech/wiki/HadoopMapReduceFirstStep