Can Redis be used as a Primary database?
Vložit
- čas přidán 27. 07. 2024
- This episode of the backend engineering show is sponsored by my friends at RedisLabs. I’m going to break this video into three sections, we will first define features that qualify a primary database? Then we will see if Redis actually check the boxes of a primary database and finally we explore the features of Redis that take it beyond a primary database. I was personally surprised by most of those.
Addendum: Redis also provides strict serializability through RedisRaft project. RedisRaft has passed the Jepsen tests, an industry standard for verifying consistency of distributed database systems.
Read more on RedisRaft here
jepsen.io/analyses/redis-raft...
Redis Raft github
github.com/RedisLabs/redisraft
Strict Serializability
jepsen.io/consistency/models/...
Chapters
0:00 Intro
1:00 What Qualifies a primary database
3:00 Does Redis Check the boxes?
7:40 beyond Redis
Resources
Try Free: bit.ly/3hWr1Uj
Redis Advantages: bit.ly/3ztx2xw
Martin Fowler talking about Impedance Mismatch: bit.ly/36ZEOD8
Transactions: bit.ly/3wTMKAw
ACID 0.5 MM Ops/Seconds on AWS: bit.ly/3ruMB5s
Consistency and Durability: bit.ly/3wYNLr8
Watch and Rollbacks: redislabs.com/blog/you-dont-n...
Redis Enterprise redislabs.com/redis-enterpris...
redis.io/topics/transactions#cas
redislabs.com/blog/you-dont-n....
docs.redislabs.com/latest/rs/... - Věda a technologie
Note regarding consistency @5:20: Redis also provides strict serializability(Strong consistency) through the RedisRaft project. RedisRaft has passed the Jepsen tests, an industry standard for verifying the consistency of distributed database systems.
Read more on RedisRaft here redislabs.com/blog/redisraft-new-strong-consistency-deployment-option/
Hey Hussein! Can you make a video on Jepsen Tests, how are they conducted?
redis raft out yet ?
"Is Redis amazing?" proudly sponsored by Redis.
Disclaimer: I use redis everyday and love it.
Cool stuff, Hussein.
But, every time you talk about some technology, you always say that everyting has its costs. Every technology has pros and cons.
But that time you only talked about pros.
I think this video is more like advertisement rather than regular discussion.
But anyway, thank you for amazing videos.
It would be great if you make another video about drawbacks of using Redis as a primary database.
I wholeheartedly agree! Being sponsored might have a contribution to this. I know one disadvantage is that it’s all in memory thus your entire database must fit in memory.
@@Squawkize memory mapped files can store more data than RAM, but definitely may impact performance of the department. What's really missing is uniform queries with optimisations. But yes, agree, in most cases can replace main service store.
@@Pifagorass can you elaborate more on what do you mean by uniform queries with optimisations?
My guess, the drawback is the price. The same reason memory is split into: CPU L1 cache, CPU L2 cache, main memory, and hard disk. price vs performance reason.
Bobosher are you uzbek?
Very nice collation and explaination of insights on Redis, thanks Hussein!
There is so much content for software engineering on you tube, but yours is on a very different level, it may sometimes have less likes but the knowledge level and quality is higher than any other content available online.
Amazing amazing videos ....your content is very crisp and clear 👍👍 ... please keep on updating new videos ...
Just what I needed... I was literally just searching for this...
Good video Hussein but, you still need to be balanced and explaining the pros and cons. Benefit and price.
When discussing choosing a database, you often dont consider the query language capabilities . Different databases has different query languages to suit different needs.
How come you ignore it so often?
Hussein, I'd be interested in hearing your thoughts on FaunDB and thank you for covering Redis as a primary database too
How would you do joins, aggregations, and query optimisation?
I believe you can use Redis as a primary database BUT the circumstances depends on what you need. AOF and snapshot can persist data yea but you’re limited to the ram you have available (which can be expensive). Not only that but less frequently visited data should be in your hd/disk instead of memory since that’s a waste of precious ram being used up. It’s a lot cheaper storing it in hd/disk.
Would be cool if Redis could store less frequently visited data to the hd/disk in exchange for some loss in speed and once it starts getting many accesses move the data from hd/disk to RAM when needed. Not sure if the loss in speed would be significant implementing this though.
Amazing in depth video ❤🎉
I used redis as cache, good to know it can be used as dB also. However, is there any configuration to use it as hot, cold or glacier type?
I almost mistaken this fireship video lol, but thanks Hussein!
Very high quality
Very useful, Thanks !
Hussein, can you schedule a video about limitations and performance benchmarking concerning (in-memory db vs buffer pool) since in both paradigm the memory is involved..
I didn’t know if redis can do so much…..but I felt like you’re the redis developer advocate 😀
Nice video. We recenly did have to use redis for some use cases as the primary source of data. It can work for simpler use cases but falls apart when it comes to slicing and dicing the data. Imagine replacing the "students" table in postgres with data in redis. Say each record contains students name, age, and city. You want to be able to fetch all students whose name starts with "xyz" AND are b/w 12-14 years of age AND live in Vancouver. You will basically end up using 4 different data structures and and a lot of overhead. Store age as a sorted set where weight is the age and the value could be the all student ids. Then store city in another Hash where key is the city and values could be ids. and so on. And then try to fetch all ids and do an intersection in memory and then look up the object with that id. Managing and extending such a system will be a nightmare. :D
I had a similar requirement once and I ended up keeping the data in memory of the application itself which was performing way better than using redis while still being maintainable.
I use Redis as the only database for a toy project. I model relations as sets. (after all, the mathematical definition of a relation is expressed as a set, right?) It works and the performance is great. However, it's quite challenging to maintain data and indexes using only bare Redis' data structures (though possible with transactionned pipelines). I did it for fun, and it was. But, relying on SQL is much easier IMO!
It's also possible to rely on "hexastore" with zsets to express any complex relations, however, I didn't need to for my app.
As soon as your data start conforming to a normalized or even semi-normalized form with standard data fields per set-kind / "entity", the argument for Redis as a primary DB goes out of the window. RDBMses are highly optimized for this kind of work and with the right kind of caching strategies and query optimizations, the "Redis as a primary DB" argument becomes laughable at best and punishable-by-flagellation at worst.
Maybe three things to think about are 1) what's the literal cost of I/O in memory vs persisted storage on disk needed 2) what's the average and foreseeable max space needed and 3) what's the probability of memory going down or being corrupted vs disk, and what are you willing to tolerate before resorting to backups (or a secondary db)
I work with in memory data models built in MSSQL and I have an amazing T-SQL experience thus far & only having tiny hiccups around querying data from external databases and combining datasets with the in memory tables. Can anyone recommend a good t-sql client for Redis?
Would be intresting to hear some thoughts on keydb
Also important, the types. Redis doesn't have booleans for example, nor does it have support for geolocation pluggins like postgis with postgres
I think redis drawback is the price. The same reason memory is split into CPU L1 cache, CPU L2 cache, main memory, and hard disk, price vs performance reason. any thoughts?
This video felt more like an ad than a discussion like your usual videos.
I have seen redis as cache. Never thought of it as primarily database. Can it be horizontally scaled without downtime?
Short answer Yes.
Long answer: No.
Realistic answer: It depends. what is your current configuration? Redis offers something called as a "Redis Cluster" which is a piece of art (coming from someone who wanted to scale Postgres) but it is some work not every library supports its 100% and / or you have to use different methods to interact with it. Also, you have to add nodes in groups i.e. suppose you have 2 replicas per master, you need to add (horizontal scale) 3 nodes at a time (1 writer and 2 replica nodes).
Although it has Primary Database features however it cannot be used for highly OLTP applications due to its serializable trx isolation level
in a multi master deployment, the purpose of which is to shard data, what if a master fails. Is there automatic failover to some replica?
I have a question, when we say it supports isolation , then does having watch violate it ? If my understanding about watch is correct, then watch will watch if there is any there operation that is trying to update the value I am referring, it will notify and probably fail the transaction. So meaning one transaction is knowing about the other transaction.
Can you clarify ?
Not recommended , if there is a lot of data , you end up paying lot of money for redis compared to other DB's
6:25 I am confused if it should be availability concern or reliability concern when the write fails with both master and slave replication?
Hey Hussain what is your thoughts on GitHub copilat?
What about relational data, cascading and foreign key checks? If we use Redis as a primary database, we will have to do all of these things in the Application level, right?
Yes you're correct, that's the price you have to pay for
Key-value NoSQL databases aren't suitable for joining in the first place. Usually the data is stored in a de-normalized fassion to prevent joining.
In redis they can still be handled at the database level though with LUA scripts. So instead of clients manually doing join, they can call a pre-cached script in the server that does the joins for them.
@@BHAVYAMATHURBEE How do you handle the case when a product name is changed for example? You need to update all the orders and change the product name for each one?
This is an age-old NoSQL vs SQL paradigm question and not really about whether Redis is a Primary DB or not. In NoSQL DBs (Redis, MongoDB, etc), you model things differently where you optimize for speed over normalization.of data. Here you may embed product name into all the orders so you don't need to do slow Joins for every read (i.e. in a denormalized form). So if you are reading the order details millions of times, you don't do millions of slow Joins. On the other hand, if at some point the product name changes, you run a script to update the product name in all orders. So you end up with millions of fast reads and a few slower updates and overall you come out ahead. BTW, there are multiple, well established patterns to handle these sorts of SQL versus NoSQL questions.
@@rajaraodv Yeah I also agree with this, you have to model your data differently
But since we know that redis is an in memory database which means all the data will stored inside the RAM itself. Don't you think the pricing for this will be way higher?
Redis can actually persist to disk. Visit their site and you will be surprised.
Never used Redis but it sure looks interesting. Is it good for modelling relational data? What are the downsides to Redis compared to say Postgres?
You can say redis is a data structure driven database. For relational data, a lot of operations need to be performed based on the use case. Postgres and redis are very different for doing a direct comparison of features.
Do a basic reading about redis, to get a good idea about it and think in this way "is it applicable in my application use case or not"
Redos makes a good light weight event broker for micro services. This is particularly true at the edge.
How does keydb compare to redis? It has master + master + slaves configure option. I can write in Master1 and if it fails then I will write Master2. Will keydb sponsor next video?
Thanks Nasser Hussein
Since Redis is executing queries in single thread, why we even need optimistic locking?
Thanks!
We used Redis a primary database for one high performant requirement
Whether Redis can be used as the primary database or not depends on what you need from your primary database.
Redis is already better than a few other NoSQL DBs and heck, even "Postgresql is a better Mongo than Mongo".
However, if your persistence requires RDBMS, then Redis is not that good of an option and honestly, it doesn't need
to be. Redis is already awesome and can be used as a Swiss army knife while building your next application.
PS. People who're complaining about the "advertisy" tone of the video should just stop. How else do you think
he'll be able to pay for stuff and keep producing quality content?
why is a cache considered costly compared to a primary database? if we store everything on redis, wouldn't the COGS overshoot?
Great video Hussein. Thanks a lot. I just have one question though mate. At 4:25, there is just one word that my colleagues (Japan) can't make out.
"The ?????? that you write ..... "
What is the ?????? word? Please let me know, if you have the time. Thanks so much.
power?
Can you explain how Isolation differs from Concurrency Control? It seems to me like Concurrency Control is part of Isolation
Isolation levels are related to the transaction you're executing.
It helps you to isolate values being used during transaction from ones in actual database.
Concurrency control is related to the values in database.
Dude Redis can even do geo stuff. Redis rocks!!!
What about data loss? In memories have a huge danger of data loss in cloud
So, will there be a RERN stack soon? 😁
the only down side is it doesn't have a query language.. which make a things much easy
Good content as always Hussein! You do not mention query fetures which are a key feature for a DB. For example how can I store a list of people in Redis and get only the people with age >18, I am supposed to get all keys of readis and cycle trough them in my application in order to filter them? How about paginating the results?
It is possible by means of sorted sets. Considering your example, you could add the records to a sorted set using the ZADD command redis.io/commands/zadd, where the score is the age and the member is the person ID, and then you could select a range utilising the ZRANGE command redis.io/commands/zrange. It basically means that you create indexes manually in Redis.
Redis is not your typical database, one needs to understand it's data structures and design their solutions around them.
High Quality.
It was 2013 that I built a tiny in-memory database and query system with Javascript. I lost interest in its development in couple of weeks as it seemed to me of the concept lacking serious applications. ... It was certainly not a good decision.
"If you are using Redis in a very write-heavy application, while saving an RDB file on disk or rewriting the AOF log Redis may use up to 2 times the memory normally used." This means you data set has to be smaller than 50% of memory if using RDB. Point in time restore? I believe is no, other than restoring truncated AOF files. The documentation has always been awesome.
I don't think that redis can be used as a primary database cuz redis works at the rams what if the server is down would my data be lost, how even i can maintain my data if the server is down ?? I don't think that redis data persistence techniques is enough and I can't take a snapshot of the data every second
Please make a crash course for using Redis as a primary database
Check out Redis University
Redis is very feature rich, but that comes with a hight cost.
I built an inventory system with redis as primary database.
It require a lot of skills and logic in programming.
The first build, i have to delete whole database logic and start again.
You should very well in reading document and take time think about it before coding
Pls I need guidance. I want to build system with redis as primary database
@@emmanuelemeneke8927 just start with simple online retail store website. You will learn a lot while building it.
Of course it can. But should we?
a new camera tech eh?
Looks more sharper, right
Does ElastiCache-managed Redis support all the features discussed in this video?
I think Redis team has gave you a script and you aren't allowed to say beyond that script. Any way nice video. Video title can be change to All Pros of a Redis in 12 minutes.
A primary database... that costs an arm and a leg
The answer is probably yes, but that depends on how much are you willing to pay
Great video man. Keep making epic stuff
Will appreciate if you can change 'master slave' to 'active passive' terminology
A few carefully written atomic Lua scripts is all we need to replicate all the most needed regular database features!
Ans:
Yes, if it makes easy to do cache-invalidation.
:)
love you Hussain sir.
Didn't felt like your honest opinion..More like an advertisement.
Dude great video with awe info. Let’s make it last longer by not talking about master & slave. It’s 2021.
Thanks!