A trick for safely using Golang Maps more efficiently
Vložit
- čas přidán 25. 03. 2024
- In this video I’ll show you how you can make your concurrent applications actually more concurrent safe by using sync mutexes and vertical sharding.
🌱 Check the "Golang Essentials" Playlist here: • Advanced Golang
👉 Join the private community to level up as software engineer: selfmadeengineer.com
📢 We're building a Discord community, come and join
/ discord
Hope you liked and thanks for watching!
#golang
Video titles ideas (for the algo):
Golang maps explained
Safely use golang maps
Concurrency problems in golang maps
It would be great if you could please do a follow up video demonstrating the actual performance improvements of this solution using benchmarks.
A few notes:
* You should add generics to make this structure reusable. That would be a cool follow up video.
* From my experience in production environments, sharding is not a silver bullet. Often hotspots develop and you should carefully examine your access patterns, in many cases a sync.Map will perform better (for single write high read cases in a cache)
* I'd advise people to find existing implementations of these types of structures rather than re-implement them, just because there can be some nice advantages to other ones.
Source: been using go professionally for >5 ys
hey sir can I have your gh for reference?
@@rafael.aloizio1769 this doxxs me so you need to give me an email or something to send it to. but there's not much interesting on it, my work is closed source unfortunately
Saves me from having to use some external solution for my structured data,, thanks for saving me from a possible data race!
Very helpful content
Thanks!
in a production ready golang app map needs three things at the same time: 1) it should be thread safe 2) auto or manuel shrinking feature. otherwise memory leak problem will arise 3) we need a range method for iterating over key and values
How would you implement shrinking the map?
Boa dica!
Thank you for sharing this content with us. Be blessed.
Thank you for the kind words!
Very nice content, clear practical examples.
I still in two minds if some things should be explained more or not, but in current times it is not necessary i think.
I don't know much about Go but wait group interface still perplexing me 😅
Hey! Glad you liked it!
Yhee, I have a tendency to make videos a bit huge so I'm making some smaller now so I can give as much value in a short amount of time. Sometimes it means cutting corners.
However one of the next videos next month will be on Golang interfaces!
@@TiagoTaquelim In times of chat gpt & Internet in my honest opinion cutting corners don't lower the value of content, people who are really interested easily obtain missing parts of puzzle and will appreciate sticking to the main topic
@@AK-vx4dy Totally agree with you!
Muito bom. Obrigado.
💪
What about having locks on each value in the map, and also having a read lock on the whole map when you iterate the keys? Seems simpler and there's less lock contention when accessing values within a certain range. I'm also a bit confused on how you're using the sha1 to create an index into the shards, that doesn't seem like it would give a fair distribution between shards. Would that be a case for consistent hashing instead? Also you're only taking the first byte from the sha1 which means your shards have a limit of 256 which I think should be mentioned here and also plays into how the distribution would be wonky.
Hey! Thanks for suggestions! I think a lock for each value could introduce additional complexity and overhead. Specially if the map has a lot of values. I think the current implementation provides a good balance between concurrency and simplicity.
For the key hashing, I agree.
More than 256 shards? Really? 😅
Very true hahahaah, I'm just honestly just asking dumb questions because I love to see how people answer them.
Communicate to share memory; don’t share memory to communicate.
👍
What's your colosheme?
Gruvbox
Why not use sync.Map for that?
Hey! Thanks for sharing I didn't knew that existed.
That is basically an implementation of what I've done here.
@@TiagoTaquelimnot exactly, sync.Map works in a completely different way, very bizarre and complicated. Your solution is interesting nonetheless, thanks for sharing it ;)
@@sealoftime Oh I see! Thanks a bunch for sharing I'll be reading on it!