Apache Kafka - Installing Kafka in Cloud

Sdílet
Vložit
  • čas přidán 6. 03. 2018
  • Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
    forms.gle/Nxk8dQUPq4o4XsA47
    -------------------------------------------------------------------
    Data Engineering using is one of the highest-paid jobs of today.
    It is going to remain in the top IT skills forever.
    Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
    I have a well-crafted success path for you.
    I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
    We created a course that takes you deep into core data engineering technology and masters it.
    If you are a working professional:
    1. Aspiring to become a data engineer.
    2. Change your career to data engineering.
    3. Grow your data engineering career.
    4. Get Databricks Spark Certification.
    5. Crack the Spark Data Engineering interviews.
    ScholarNest is offering a one-stop integrated Learning Path.
    The course is open for registration.
    The course delivers an example-driven approach and project-based learning.
    You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
    The course comes with the following integrated services.
    1. Technical support and Doubt Clarification
    2. Live Project Discussion
    3. Resume Building
    4. Interview Preparation
    5. Mock Interviews
    Course Duration: 6 Months
    Course Prerequisite: Programming and SQL Knowledge
    Target Audience: Working Professionals
    Batch start: Registration Started
    Fill out the below form for more details and course inquiries.
    forms.gle/Nxk8dQUPq4o4XsA47
    --------------------------------------------------------------------------
    Learn more at www.scholarnest.com/
    Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
    ========================================================
    SPARK COURSES
    -----------------------------
    www.scholarnest.com/courses/s...
    www.scholarnest.com/courses/s...
    www.scholarnest.com/courses/s...
    www.scholarnest.com/courses/s...
    www.scholarnest.com/courses/d...
    KAFKA COURSES
    --------------------------------
    www.scholarnest.com/courses/a...
    www.scholarnest.com/courses/k...
    www.scholarnest.com/courses/s...
    AWS CLOUD
    ------------------------
    www.scholarnest.com/courses/a...
    www.scholarnest.com/courses/a...
    PYTHON
    ------------------
    www.scholarnest.com/courses/p...
    ========================================
    We are also available on the Udemy Platform
    Check out the below link for our Courses on Udemy
    www.learningjournal.guru/cour...
    =======================================
    You can also find us on Oreilly Learning
    www.oreilly.com/library/view/...
    www.oreilly.com/videos/apache...
    www.oreilly.com/videos/kafka-...
    www.oreilly.com/videos/spark-...
    www.oreilly.com/videos/spark-...
    www.oreilly.com/videos/apache...
    www.oreilly.com/videos/real-t...
    www.oreilly.com/videos/real-t...
    =========================================
    Follow us on Social Media
    / scholarnest
    / scholarnesttechnologies
    / scholarnest
    / scholarnest
    github.com/ScholarNest
    github.com/learningJournal/
    ========================================

Komentáře • 43

  • @ScholarNest
    @ScholarNest  Před 3 lety

    Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
    www.learningjournal.guru/courses/

  • @pranavsindhavad8654
    @pranavsindhavad8654 Před 4 lety

    Must say Prashant has an amazing knack for teaching and making complex topics more simpler and understandable to all audiences, please keep up the good work Prashant, wish you all the best and keep it coming.

  • @foodandsunshine5090
    @foodandsunshine5090 Před 6 lety +3

    Very good videos. Loved it. The sound quality is good as well. Thanks Learning Journal!

  • @AshishGupta-cb8hd
    @AshishGupta-cb8hd Před 5 lety

    Man, you are awesome....please keep sharing the knowledge

  • @carloslfu
    @carloslfu Před 5 lety +1

    Great stuff man! Thanks!

  • @meirzeevi3618
    @meirzeevi3618 Před 5 lety

    One of the best tutorials
    Thank you for sharing

  • @DataPro360
    @DataPro360 Před 3 lety

    great video, straight to the result.

  • @appy2501
    @appy2501 Před 5 lety

    nice work sir - great work sharing information :-)

  • @codewithsalman
    @codewithsalman Před 4 lety

    very informative video and channel .

  • @epicfoods2891
    @epicfoods2891 Před 4 lety

    great video

  • @Jay-ib2ii
    @Jay-ib2ii Před 6 lety

    Hi sir, your videos are amazing. Could you also make a tutorial on Apache kafka integration with Apache flink and streaming it.?

  • @neeharikamandarapu8517

    Thank you so much sir for the Kafka tutorial. Could you please do a tutorial on integration of Kafka with Spark streaming

    • @ScholarNest
      @ScholarNest  Před 6 lety

      Yes, that will be part of my upcoming Spark streaming playlist.

  • @MrManishaman
    @MrManishaman Před 5 lety

    Sir i am unable to start kafka-server it says permission denies. I am using RHEL server in my local machine

  • @balaji3229
    @balaji3229 Před 5 lety

    Sir iam a beginner in Kafka. I need to implement it in my project.
    We can expect some 200k records in the mysql user table. I have to track User events like User Creation, User Address updation, User Email address updation etc....
    I have to integrate Kafka producer here. The important requirement is we have to maintain the ordering of events.
    I have 2 clarifications sir,
    1. Is it to good to have one topic called Users with multiple partions based on the Users role. The idea of having more partitions is to make the solution more scalable.
    2. How many Kafka brokers we can have in Accepatnce and Development environment to make this an optimal solution.

  • @saurabhs619
    @saurabhs619 Před 5 lety

    can I use AWS instead ?

  • @rikuntri
    @rikuntri Před 5 lety

    Hi sir, can you please make a video on how to install various big data components in Google data proc e.g nifi or flink

  • @vibhorgupta5187
    @vibhorgupta5187 Před 6 lety

    Hi
    can you please make a video on..how to setup kafka in cloudera quickstart vm. That will be really helpful.

  • @nationviews6760
    @nationviews6760 Před 6 lety

    thank you so much, Sir, for this video. please, can you Explain how we will take data from other sources like Sensor data and how we will use that data to get the reliable result through Spark Streaming?

    • @ScholarNest
      @ScholarNest  Před 6 lety

      That's a great question but hard to answer as a comment reply. Sensors are programmable devices. You can push simple program to your sensor devices. These programs can send data to an IoT gateway over MQTT or HTTP protocol. All cloud providers like Google and AWS, both of them offer IoT gateway service. Once data reaches to IoT gateway, you can configure the gateway to pass it to Kafka. That's it. Rest is Spark streaming over Kafka. I will create tutorials on this last part. I can show a demo on first part also but I don't think I have funds to invest on that.

  • @sushmashamsundar8219
    @sushmashamsundar8219 Před 5 lety

    sir am getting timed out exception while creating topics. , no solution in goole also. pls help.
    Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING

  • @hussamcheema
    @hussamcheema Před 3 lety

    I'm getting this error on the fist command:
    WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092
    ) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient).

  • @eliavmaman3203
    @eliavmaman3203 Před 5 lety

    Hey Great tutorial. i have a question: i have a Node js server configured to the Google cloud Kafka server
    and in the url i put the External IP and i get an error 'host resolution failure', what could be the reason?

  • @SahilRallySuperStar
    @SahilRallySuperStar Před 4 lety

    1. We can have a consumer facade in the consumer group which talks to Kafka . The only responsibility it has is to distribute workload among other consumers. This way we can autoscale consumers ?
    2. Don't we have autoscaling mechanism for Brokers ? Seems like we need to provide number of brokers at the start of server. ???

  • @veerukbr1184
    @veerukbr1184 Před 6 lety

    What is the difference between Kafka streaming and spark streaming ?
    How to stream customized website data using Kafka ??

    • @ScholarNest
      @ScholarNest  Před 6 lety

      Windows phones were good. But we failed them and embraced Android. That's called IT consumerization. Kafka stream is graduating to become a Spark streaming competitor. It's we, developers and architects who are going to choose a winner.

  • @anurag17091977
    @anurag17091977 Před 3 lety

    I like your videos, I need Kafka version is 2.8. Please share link for same.

  • @sharanappagr7175
    @sharanappagr7175 Před 4 lety +1

    When i ran the command got below error kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
    Exception in thread "main" joptsimple.UnrecognizedOptionException: bootstrap-server is not a recognized option

    • @surbhiagrawal3951
      @surbhiagrawal3951 Před 4 lety

      i am also getting same error ,,, can you guide how to resolve

  • @vijaymahendrasinghrana1241

    Any guidline or documentation to install kafka on openshift3.11.

  • @shivareddy5717
    @shivareddy5717 Před 5 lety

    Where does the messages get stored in the consumer ?

    • @ScholarNest
      @ScholarNest  Před 5 lety

      Messages are stored at the broker in the log partition segment files.

  • @NishantKumar-xx5bq
    @NishantKumar-xx5bq Před 4 lety

    i am using kafka 2.1.1 on GCP but i am unable to create a producer. i am getting below error.
    "zookeeper is not a recognized option". i tried with both zookeeper and bootstrap-server. it didn't work

    • @rajeshdamam8861
      @rajeshdamam8861 Před 4 lety

      Try this one:
      kafka-1-vm:~$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testCreated topic "test"
      kafka-1-vm:~$ kafka-topics.sh --list --zookeeper localhost:2181
      test
      Starting producer and consumer both:
      kafka-1-vm:~$ kafka-console-kafka-console-consumer.sh kafka-console-producer.sh
      kafka-1-vm:~$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
      >This is a test message form Raj
      kafka-1-vm:~$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
      This is a test message form Raj

  • @babujayaraman7810
    @babujayaraman7810 Před 5 lety

    we have to change the command to kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test if you use version older than 2.2 you should use --zookeeper option and pass connection string to zookeeper

    • @surbhiagrawal3951
      @surbhiagrawal3951 Před 4 lety

      thanks my error is resolevd through your command ..i have created a topic

  • @MohammadFaizanKhanJ
    @MohammadFaizanKhanJ Před 5 lety

    Can't we use Apache Kafka on windows 10?

    • @ScholarNest
      @ScholarNest  Před 5 lety

      We can. I have a video in Kafka Streams playlist.

  • @cubbybear8322
    @cubbybear8322 Před 4 lety

    This was going great untill i realised that you get error with every command you try:( e.g. i cannot create a topic,i get an error every time.Please help!

    • @yassinebenhadi4883
      @yassinebenhadi4883 Před 4 lety

      Here is the fix: stackoverflow.com/questions/55494988/why-is-kafka-not-creating-a-topic-bootstrap-server-is-not-a-recognized-option