Hadoop Installation on Windows-10 Natively | Light & Fast
Vložit
- čas přidán 6. 09. 2024
- Hello Guys,
In this video i have explained in detail, how you can Setup a single node hadoop cluster on you native windows 10 PC.
Many of us are very comfortable with Windows environment, or may be do not have heavy resources on a computer or a laptop, this setup will help you to setup a Hadoop environment, Natively on you windows 10 computer, which will be very light to use, and there will be no longer any need to run those heavy Hadoop Virtual machines.
Jump Timings
=================
Why Windows? : 00:16
Benefits of custom setup : 00:40
Host laptop configuration : 01:50
Check JDK : 02:40
Environment Variable Setup: 03:59
Hadoop Installation: 05:20
Config Files Setup : 07:15
Winutils usage and setup : 12:55
Testing the environment : 14:55
Checking URLs: 16:25
Important links (mentioned in the Video)
=========================================
Hadoop Download : hadoop.apache....
Winutils Download: github.com/sel...
YARN UI (Post setup): localhost:8088/cluster
NameNode UI(Post setup) :localhost:9870
Property Tags (mentioned in the Video) Replace () with angle brackets in below tags
=======================================
core-site.xml
------------------
(configuration)
(property)
(name)fs.defaultFS(/name)
(value)hdfs://localhost:9000(/value)
(/property)
(/configuration)
mapred-site.xml
---------------------
(configuration)
(property)
(name)mapreduce.framework.name(/name)
(value)yarn(/value)
(/property)
(/configuration)
hadoop-env.cmd
--------------
replace %JAVA_HOME% with the path of the java
C\java\jdk1.8xx (without bin)
hdfs-site.xml
--------------------
(configuration)
(property)
(name)dfs.replication(/name)
(value)1(/value)
(/property)
(property)
(name)dfs.namenode.name.dir(/name)
(value)xxxxxx\data
amenode(/value)
(/property)
(property)
(name)dfs.datanode.data.dir(/name)
(value) xxxxxdata\datanode(/value)
(/property)
(/configuration)
yarn-site.xml
------------------
(configuration)
(property)
(name)yarn.nodemanager.aux-services(/name)
(value)mapreduce_shuffle(/value)
(/property)
(property)
(name)yarn.nodemanager.auxservices.mapreduce.shuffle.class(/name)
(value)org.apache.hadoop.mapred.ShuffleHandler(/value)
(/property)
(/configuration)
Note:
In case you notice any error like
JAVA_HOME not correctly setup up - While using/starting hadoop ,
replace your "C:\Program Files\xxx" to "C:\Progra~1\xxx" in your JAVA_HOME variable. (the space might cause this issue.
This video was so comprehensive , Perfect , all the points covered in the video + comment including resolution for the "path not found error" , Thanks for this video MR. Perfect
Hi sir good evening i have installed hadoop as per your instructions in video , but while checking
i am getting this : " SHUTDOWN_MSG: Shutting down DataNode at DESKTOP-8EVNK70/192.168.233.1 " for shutting down Datanode and " SHUTDOWN_MSG: Shutting down NameNode at DESKTOP-8EVNK70/192.168.233.1 " for shutting down Namenode ,, And same message is comming for NodeManager as well like : " SHUTDOWN_MSG: Shutting down NodeManager at DESKTOP-8EVNK70/192.168.233.1 " , So can you fix it ? or tell me what to do ? i am really very much tensed because of unable to practice ,,
Guys if you face problem like "Java_home not correctly setup",just move your Java folder out of program files to C: and edit path accordingly...it works fine after that!
Worked fine after your advice. Thanks :)
Thanks bro :-)
Thanks :))))))))
I am getting this error.please help
HADOOP_HDFS_HOME not found
When I type command start-dfs.cmd Namenode is shutting down, so what I do?
Hi, the namenode UI doesnt work on 9870, but i am able to run yarn UI on 8088 why is that?
error in resource manager and node manager while start-all cmd
how to solve this
Sir ,
My Hadoop version 3.3.0
Successful install Java and Hadoop.
But start only namenode, others are shutdown.
What's a solution for this?
Pls reply
Hey there.
I think its better if the dir path in hdfs-site.xml is mentioned as -
dfs.namenode.name.dir
file:///C:/hadoop-2.7.2/data/namenode
dfs.datanode.data.dir
file:///C:/hadoop-2.7.2/data/datanode
instead of using the Windows path.
Otherwise, it results in MapReduce jobs to fail (eg. running HIVE insert query, etc) with below exception - -
"CreateSymbolicLink error (1314): A required privilege is not held by the client."
I didn't find winutils for hadoop version 3.3.1 , can you help guys.
Thanks you. It worked for me.can you please provide link for spark,scala, kafka setup.
Good going and thanks for sharing.
hello, my namemonde and datanode are not working. yarn is working. and the resource manager is working.but he i use the jps command there is no namenode or datanode.
Good explanation. Please upload spark
Thank you! The best of all!
localhost:9870 or localhost:50070 not working for namenode
Hi, Thanks for the video, do you have continuation of this one,spark on yarn integrate in with Hive in like you mentioned in end of this video..Please share link
Please mention that the start-all.cmd, or start-dfs or start-yarn must be run from a cmd-window with Administrator privilege.
Thanks a lot
Hello, I have followed every step of yours without fail. But finally when I check Hadoop version on cmd prompt., Hadoop cmd has not found. Hadoop is not Installed....I need help... Very urgent sir... Please reply within hours🙏🙏🙏🙏
Download the winutils from Microsoft shell directly and then be careful while setting environment variables. The paths should not have space in them. Java by default gets installed in Program Files. Replace "Program Files" with "Progra~1" and "Program Files(x86)" with "Progra~2" in the path of the variables.
@@nabamitachakraborty2069 thank u. I will try
its really cool.thanks man
hello, nice tutorial but i have a problem, i use hadoop version 3.1.3 and yarn ui does not work for me in localhost 8088, but name node work on port 9870... what happend?
everything else working fine for me,but nodemanager shuts down after starting...any solution?
make sure you downloaded the correct win utils version compatible with your hadoop release.
you can find all versions here: github.com/cdarlint/winutils
@@rajaekribii7729 I am using Hadoop 3.1.3,but cant find utils for that version on github!
@@rajaekribii7729 I downloaded everything from the links provided in video description...in video he downloaded hadoop 3.1.3 and winutils for hadoop 3.1.1 , that worked for him.
Please I need help. I ran the hdfs namenode -format (perfect)
Now when I run the " D:" it says "The device is not ready. Please I need help to fix this
#(the port: 8088 does not work with me)
im getting error can any one please tell me how to solve it!!!!
and i cant able to copy file to hdfs
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\ashwin>hadoop version
The system cannot find the path specified.
Error: JAVA_HOME is incorrectly set.
Please update F:\hadoop\hadoop-3.2.2\etc\hadoop\hadoop-env.cmd
'-Xmx512m' is not recognized as an internal or external command,
operable program or batch file.
you are awesome bro
Error message :
:\Users\mahesh_timothy>hadoop version
Error: JAVA_HOME is incorrectly set.
Please update C:\Bigdata\hadoop-3.1.3\etc\hadoop\hadoop-env.cmd
'-Xmx512m' is not recognized as an internal or external command,
operable program or batch file.
localhost:8088/ isn't working. It'd be great of someone can help me out.
thank you!!!
C:\Users\karthik.s>hadoop version
Error: JAVA_HOME is incorrectly set.
Please update E:\hadoop-3.1.3\etc\hadoop\hadoop-env.cmd
'-Xmx512m' is not recognized as an internal or external command,
operable program or batch file.
i got this error while checking hadoop version . i changed the java home path :set JAVA_HOME=C:\ProgramFiles\Java\jdk1.8.0_241
but still i got this error.
,just move your Java folder out of program files to C: and edit path accordingly...it works fine after that!
Thank U , But please, how can i know which port should i use in hadoop 3.1.3? (the port: 8088 does not work with me)
I have teh same problem, did you find a solution?
#(the port: 8088 does not work with me)
im getting same error can any one please tell me how to solve it!!!!
and i cant able to copy file to hdfs
Please resolve my error
i can't find winutils for hadoop 2.7.7 version., can you help me out on this..?
Download winutils.exe for 2.7.7 from github and copy it in bin folder of hadoop....This problem will get solved
Thanks for your video - I could not able to open Local host ( localhost:8088 not working ) and data node are not showing in localhost:9870/dfshealth.html#tab-overview, check the deamons they r showing messages like volumes failed: 1, hadoop\hdfs\lib\error_prone_annotations-2.2.0.jar; org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, ERROR datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1 , When stopping the service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
java.lang.NullPointerException, INFO service.AbstractService: Service NodeManager failed in state STOPPED
java.lang.NullPointerException,
Syntax error in URI , hadoop-3.1.3\data
amenode. Please check hdfs configuration.java.net.URI
SyntaxException: Illegal character in opaque part at index 2: hadoop-3.1.3\data
amenode
at java.net.URI$Parser.fail(URI.java:2847)
I have the same problem now, did you find a solution?
@@DrumRodri have you got any solution dude?
@@Jk-jk6fv sorry but not, I change to linux and works better but its a diferent instalation... I whis u luck
Sir while testing the setup it gives error 😡
Like JAVA_HOME is incorrectly set please update
How can i fix it?
please help me
Like the system cannot find the path specified it is asking to update hadoop env.cmd
But the path is in the environment variables is same in hadoop env.cmd
Hi Sir , I have followed same steps, while running below command
hadoop version am getting error .
Error message : C:\Users\mahesh_timothy>hadoop -version
Error: JAVA_HOME is incorrectly set.
Please update C:\Bigdata\hadoop-3.1.3\etc\hadoop\hadoop-env.cmd
'-Xmx512m' is not recognized as an internal or external command,
operable program or batch file.
Currently JAVA_HOME - C:\Program Files\Java\jdk1.8.0_231
HADOOP_HOME - C:\Bigdata\hadoop-3.1.3
please help me.. I tried all possible ways to my knowledge still facing issue ( i have removed spaces tried as well).
thanks in advance
This is probably happening because your java is installed in Program Files, this has space, Which might cause issue for hadoop to read the correct path.
Do 1 thing, Where your JAVA_HOME is set, in same path, replace "Program Files" with "PROGRA~1" let all other slashes and path be same.
Save the environment variables, and restart your PC.
@@GKCodelabs Hi Sir . Thanks a lot for helping out.. as suggested i have replaced "Program Files " with " PRGORA~1" and it worked fine... now... I have few questions.
with the above video, we have installed Hadoop. I have seen some of your other videos where you have mentioned that setting up few applications ( like Intellij for spark development , Ozzie job scheduling , setting up GIT and Git hub for intelli J) .. Sir could you please help me with the sequence of the video list..
Can we have any one video where we process data from source to Target.. .completely on Hadoop
starting from Data Ingestion, Data Data Transformation and Data Visualization ( we can skip last - Data visualization ).
one last question if i want to become become big data developer Do i need to learn all these. . or i can concentrate on any programming language like ( Python or Scala ) is it sufficient..
If you don't mind ..can you please give your contact no to the below mail..id. . mahesh.tenapali@gmail.com.
so I can contact you based on your availability...Thank you...
@@maheshtenapali8938 Mahesh, you can put like : "%JAVA_HOME%" in Hadoop-env.cmd file. Put it in double quotes. It will work for sure.
@@tusharpatil9289 Thanks Tushar. I have replaced "Program Files" with "PROGRA~1". It worked fine. . Thanks for your input :)
,just move your Java folder out of program files to C: and edit path accordingly...it works fine after that!
I am getting the following error during start-all.cmd. Please advice
Error occurred during initialization of VM
Could not reserve enough space for 1024000KB object heap
1/1 log-dirs have errors: [ E:/software/hadoop/hadoop/logs/userlogs :
Directory is not writable: E:\software\hadoop\hadoop\logs\userlogs ]
and localhost:50070.not connecting......please help me
i am getting this error "Could not find or load main class" please help
Open your hadoop-env from /etc/hadoop and edit that. At the last statement remove %username % and replace any string. It will work
@@Jk-jk6fv worked for me, thanks a ton