What is a Data Lake?
Vložit
- čas přidán 25. 06. 2024
- Learn more about watsonx: ibm.biz/BdvxDa
A data lake is a centralized repository that allows you to store a vast amount of raw data in its native format until it is needed. Data-driven businesses often use this storage architecture to get more business value from their data assets.
Check out this quick overview video with Adam Kocoloski as he goes through a data lake architecture and explains how data lakes can empower you to collaborate and analyze data in different ways that will help you make make smarter and faster business decisions.
#DataLake #CloudStorage #IBMCloud - Věda a technologie
I was way too distracted by the way he was writing in reverse. I was switching between being in awe as how well he was writing in reverse, and his explanation. Thus, I have to watch it again after accepting the fact that he is a genius, and then finally get to understand everything. Great explanation.
I think he was writing normally and the video was flipped/mirrored afterwards.
He writes normally on a clear glass screen, with a camera on the other side recording it in reverse - then flips the image. Still pretty genius.
Amazing that you can write mirroring letters ....
Read this post on our community page that explains it all ibm.co/2SA1vGd
You know you can invert in post processing right? Lol
@@hdebbache2000 He is actually left handed and married in real life but not in this video jajaja
@@hdebbache2000 But it would not have the same impact...for sure, all of viewers take a look on that and said: hes good!
I think the concept of data lake should have just focused on explaining the "Store" box.
Great explanation, thank you!
Very clear and helpful. He writes like Ira Joe Fisher
Great presentation, thank you!
Thanks for watching Malcom!
Simple and very informative. thx alot
Need to explain this to Snr Management, this video is very helpful in breaking it down into something I can explain to others.
Interesting but Data Lake is not only used by ML. It usually used to store unstructured raw data. Some governance can be applied, however, you don’t build Dashboard out of the data lake. You first need to model that data into a Data Warehouse using dimensional modeling (allowing you to extract different dimension of your data). This multi dimension represented by few tables will allow you to slice the data in multiple ways, making reporting, thus dashboards easy to build.
This is why Airbyte/Fivetran + Snowflake + DBT are the most popular data stack on the market right now.
😯
Thanks, that was very informative
Very clear and helpful. Many thanks!
Glad it was helpful Brandon!
Man this is an awesome idea to stand behind the mirror and write it..👍🏼
Good explanation, thank you. However, talk can get started with "Big Data" - which means data lakes are intended to store, manage and serve large Big volume, variability, velocity. Data is ingested in native format. It need to be kept organized, controlled and managed - governance. Data needs to be served in native or processed further for other needs - reporting and visualization, recommendations, process automations and more. Some real-life use cases to start the discussion.
If the viewer already knows bits of data world (databases, datawarehouse, data lake etc), this helps to consolidate that understanding.
Why is it that every data lake explanation is full of theory without any concrete examples? Aren't all of us here because we're SQL or Cube programmers and want to know whats so great about Data Lakes? All I see is the same thing I do with sql databases: import the data, prep and transform it and then query it directly or create dashboard applications.
Hi Mauro, does this help? czcams.com/video/IPkQpBdde5Y/video.html
I agree. This video is very similar to what we do with SQL. He has not really told what data lake is. But, whatever he told is true about Data Lake. Here is a list of differences in Data Lake that are not possible in standard SQL based RDBMSs.
1. Big Data
2. On the Cloud (this is possible)
3. Separation of Data from Data Processing Engine
4. Self Service Model
5. ML (this can be done)
6. Data in native format (csv/parquet/json/avro/...)
All the above are common to Big Data. Here is the list of data lake differentiator.
7. Central Repository; means single source of truth.
data is really the new oil! nice explanation
You almost had me with the mirrored letters, until I realized- your wedding ring is on your right hand in this video. Very nice camera trick.
it could be on his right hand. not everyone wears it on their left.
@@b00psie yes it could be, but it most likely isnt
good explanation
Very good explanation
Thank you, Kuldeep!
do you use data warehouse at the "store" phase?
Mirroring letters was a bit spooky/distracting at first.
But great and simple content. Thanks.
ibm is still using the traditional rdbms right? what about hadoop?
awesome !!!
Great video. I posted this to my LinkedIn.
Hi Cotton Hollow Distilling! Thanks for the link love! -Sai
So does a Data Lake fall under Document Store due to it ingesting all types of Meta-Data such as Audio /Media , and text ?
I still have no idea what concretely is a data lake!
thanks
Left-handed.... It explains everything
He s actually using his right hand to write
What does “infuse” mean in this context ? I could not find an answer searching on the Internet.
Infuse to business decisions for managers (Dashboard), consume by other part of the service in an app(Application), or automate to make the entire process smarter with AI (Automation)
"Data Lake" a piece of unnecessary jargon that adds nothing to the conversation. We've been dealing with these principles for decades already.
Thank you! can you make video compare with data lake and data warehouse?
We're glad you enjoyed it! 😃 We'll pass your feedback on to our team.
Nice Fancy Arrows on your diagram
Thanks, but I don't understand how this differs from data warehouses and ETL.
Actually it doesn't, what makes it largely different is the kind of features a data lakes gives. Its catalogues data and makes it more usable traceable for external data operations. So ya u can say I can simply extract data out of my warehouse/etl system and then operationile for my spark jobs .... Chances are in a data lake solution this solution is already inbread in it with it's own ui or api for easy operationalisation ( spark job related transformation of data , munging cleaning etc) ... A data lake is a full blown solution more importantly an overlay over the existing data infrastructure u have. Maybe an onprem hadoop, or clustered mongodb. A data lake software should primarily be able to create a single view of these and make sense of it. It's a thin line but the data lakes are supposed to be more organized.....
I would say it depends on the underlying tech. Data warehouses (DWHs) and Extract-Transform-Load (ETL) is focused on relational databases (Postgres/Oracle/Microsoft/MariaDB/MySQL/SQLite), whereas a Data Lake also includes "Not Only SQL" (NoSQL) technologies like Kafka (data streams), Hadoop (Document Store/csv file storage), Impala (SQL query engine for Hadoop), etc.
When it comes to concepts, it *heavily* overlaps, IMO.
@@David-2501 I think there's a correction, DW are not focused of relational but dimensional databases
Data about data, is that same as metadata?
Yep!
E X C L L E N T !!!
Next time you do this you should write in the direction relative to yourself then just upload a horizontally flipped video.
Hi Scott...actually that is what we did :) Check out this blog post for the details: czcams.com/channels/KWaEZ-_VweaEx1j62do_vQ.htmlcommunity?lb=Ugzf5SL_yh9NglCJzgF4AaABCQ
@@IBMTechnologyThe wedding ring on the "wrong" hand gives it away. Can you post-production CGI it onto the correct hand? ;)
where the data lake ?
Hope all the window panes in your office have not become data lakes?
More like what is Business Intelligence
I dont think this actually explained the concept of Data Lake. Is it just a simple design pattern?
Why dont' you just write forwards and then flip the video?