![Shuffle Sharding](/img/default-banner.jpg)
- 14
- 26 260
Shuffle Sharding
United States
Registrace 20. 09. 2021
Videos from Colm MacCárthaigh; VP and Distinguished Engineer at Amazon Web Services. It's mostly technical deep dives!
OxE - Seattle's options - Ranked Choice Voting and Approval Voting - Deep dive and Q&A.
This November Seattle is voting on whether to change voting systems. This video is a deep dive into the three options; keeping the current system, using Ranked Choice Voting, or Approval Voting, along with Q&A at the end.
zhlédnutí: 378
Video
0xD - Safekeeping with no Combination - building and operating systems that handle sensitive data
zhlédnutí 544Před 2 lety
How can we build and operate systems where even the service provider does not have access to the data that is being managed?
0xC - Side-channels and jitter
zhlédnutí 398Před 2 lety
Why doesn't jitter or noise mitigate side-channels?
0xA - Distributed Systems Engineering: Idempotency
zhlédnutí 929Před 2 lety
In our tenth stream we get into Idempotency, a really core topic for understanding distributed systems.
0xB - A short one on Shuffle Sharding
zhlédnutí 1,1KPřed 2 lety
In our 11th series we go deep on Shuffle Sharding, the namesake of this channel!
0x9 - Distributed Systems Engineering: Idempotency
zhlédnutí 1KPřed 2 lety
In our first session on Distributed Systems Engineering, we dive into Idempotency.
0x8 - "How do I get people to do the right thing?"
zhlédnutí 1,2KPřed 2 lety
In our first Principal Engineer soft skills session we'll cover one of the most common challenges senior engineers face ... "How do I get people to do the right thing?" ... whatever that means!
0x7 - Functional Programming in C
zhlédnutí 3,4KPřed 2 lety
Our brains think differently in Functional Programming languages and that's really useful! More useful than the languages themselves. Here we go through how functional programming techniques were used in s2n, a C99 code base, when C is about the least functional programming language around!
0x6 - A deep dive into DNS
zhlédnutí 8KPřed 2 lety
A deep dive into how DNS works including full walkthroughs of name resolution, from someone who has built several DNS services.
0x5 - Shuffling and Sampling
zhlédnutí 658Před 2 lety
A deep dive into Fisher-Yates shuffling and reservoir sampling. Two easy algorithms that we can learn a lot from.
0x4 - Deep into blockchains and NFTs
zhlédnutí 537Před 2 lety
How do blockchains work? We dive into blockchains as used in crypto-currencies and NFTs.
0x3 - Side Channels
zhlédnutí 569Před 2 lety
A deep dive into side channels; what are they and how can we protect systems against them?
0x2 - Confidential Computing at AWS, the Nitro system and Nitro Enclaves
zhlédnutí 2KPřed 2 lety
A deep dive into how the AWS Nitro System provides confidential computing between AWS and AWS customers, and how Nitro Enclaves can be used by customers to build a confidential computing boundary around sensitive data processing that is isolated even from themselves.
0x1 - How AWS SIGv4 and SIGv4A work
zhlédnutí 6KPřed 2 lety
A deep dive into how web requests are signed and authenticated for Amazon Web Services
Interesting one, thank you for this!
thx!
These sub keys (hashes) are generated when a secret access key is requested and propagated to services at that time, right? The way it was explained made it sound like there was some batch job generating keys every day
The red phone is for calling Jeff Bezos when quantum computers break RSA.
very cool. And yes, DNS nameservice
It was quite helpful. Aversion to risk is aversion to change, not necessarily aversion to growth but it can be, specially when it comes to department and company culture, whether it’s lack of cohesion in team work or growth.
OMG the first 15 minutes is "functional programming sucks and i don't understand it". I hope that's not the whole thing.
Nice!
thank you
couldnt agree more when it comes to team members who prefer top submit clever (or obscure or impressive) code over simple code
This is a magnificent video and explanation about DNS and even shows you real commands to utilize.
I created a lambda with a function URL. is there a straightforward example for making a call to this URL with fetch, or jquery ajax, etc. ? ( or even axios within react )
While saying you were using FP concepts, you were describing a combination of data and functions which sounded very OO to me.
something funny about discussing cutting edge security tech and relying on holding up sheets of paper to the camera 🤣🤣
Could you link the tweet/docs you are mentioning at 39:09 please?
Thank you for the elaborate explanation.
tldw: you send your `id` along with fancy `hash(key_for_that_id)`. AWS can then look up their stored copy of `key_for_that_id` and calculate hash(key_for_that_id) on their end. As long as the two hashes match, all is well. [Fancy here means they also take region, service name, date, url into account etc]
Here you dropped your 👑
one of the best video explaining sigv4(a)!
Great stuff, thanks Colm!
better if you explain with diagram (visualization method), not easy to follow and had to stop watching and look for another resource
Given my political leanings I was happy to see Peltola win in Alaska, but my guess is this is actually an example in which RCV failed to find the Condorcet winner (would have beat either of the other two candidates in a head to head), who was likely Begich. Doesn't mean there was anything about the election that was rigged or unfair, but I don't necessarily think its a great example of RCV best representing voter preference.
Thanks for putting this together, it's cool stuff! One factor that wouldn't be relevant in a better world: it's a lot simpler to talk about the validity of an approval vote. Given that our society has reached the point of people trying to discount first-past-the-post votes based on supposed patterns in when they were counted - it makes me pretty doubtful that we'd handle ranked choice runoffs calmly and logically. That said, people don't talk enough about the benefits of those indirect impacts on coalition-building, so this was neat.
Currently curl 7.75+ supports Sigv4. It would help drive adoption of Sigv4a if curl supports it.
0x1F44D
Is that red phone behind just for decoration? :)
@@ShuffleSharding 😮
This is fantastic!
Hi Colm, 0x9 and 0xA are duplicated btw (both Idempotency). Thanks so much for taking the time to do these videos. They are very instructive at many levels.
So, why didn't you just remove region information from the hmac hash to make token work across regions? It must have been very expensive to optimize performance for public/private key encryption considering AWS scale
Because that isn’t allowed, it is not hard to gen another sig for another region
I had this in my bookmarks when it was on twitch and to my horror it was removed, luckily I found it here so I can finally watch it. Why was it removed from Twitch?
Caralho mano, o cara vez o video de bom coração e os cara fica reclamando pqp, q povo chato
At 53:07 or so. Is there a bug in the code? You aren't guaranteed to fill the list if the data_size > sample_size. I think you need to fill all the samples, then start sampling using random numbers. This manifests when the sample size is only a bit smaller than the overall data size. Sometimes the sample list doesn't fill up but if data is sufficiently large the chance a sample won't be filled trends towards 0.
Yes, I found that too. The fix is easy: until the sample set is full, add items as they come in (as explained in words, but the code does not do that part).
how to do that with laravel , or php through sdk? there’s no clear ref?
Excellent video. Excellent background. :)
nice glasses )
Is there a way to suppress sound of twitch when a user is joining or posting comment ? The sound is very distracting and sometimes we miss important keywords that you are saying because of that sound. Thanks
Really great video
Loving your work here :)
That was super informative, thanks a lot!
Nice lecture! Could you please add all the videos to a series tutorial playlist? Thanks!
Interesting
Nice!
Cool!
the noise in the background is annoying ... is that a computer?
Are you really a programmer?
Using unsigned variables for plain bool values. What a waste.
Unnecessary maybe, but a waste? Explain.
Loved the content so far. The only issue is that the video quality is 480p so when it gets to the coding section, it's really hard to read the screen, even when you make the font bigger. Other than that, great content. I missed breakfast cause I got hooked on the topic.