"Programming Distributed Systems" by Mae Milano
Vložit
- čas přidán 9. 10. 2023
- Our interconnected world is increasingly reliant on distributed systems of unprecedented scale, serving applications which must share state across the globe. And, despite decades of research, we're still not sure how to program them! In this talk, I'll show how to use ideas from programming languages to make programming at scale easier, without sacrificing performance, correctness, or expressive power in the process. We'll see how slight tweaks to modern imperative programming languages can provably eliminate common errors due to replica consistency or concurrency---with little to no programmer effort. We'll see how new language designs can unlock new systems designs, yielding both more comprehensible protocols and better performance. And we'll conclude by imagining together the role that a new cloud-centric programming language could play in the next generation of distributed programs.
Mae Milano
Incoming Assistant Professor, Princeton University
@mbpmilano
Mae Milano is an incoming assistant professor at Princeton University and a postdoctoral scholar at UC Berkeley working at the intersection of Programming Languages, Distributed Systems, and Databases. Her work has appeared at top-tier academic venues and has attracted the attention of the Swift language team. She is a recipient of the NDSEG Fellowship, has won several awards for her writing and service, and is a founding member of the Computing Connections Fellowship's selection committee (computingconnections.org/).
----
Recorded on Sept 21, 2023 in the PWLConf track at Strange Loop 2023 in St. Louis, MO.
thestrangeloop.com
pwlconf - Věda a technologie
This might be the most technically interesting talk I've seen out of Strange Loop this year. Kudos, Mae.
Agreed, my head hurts.
+1
Agreed, what a fantastic talk. I was just reading some papers about consistency models across many databases. This talk walks through it in a very cool way
This is a great talk, with an outstanding delivery. Kudos to Mae Milano.
Amazing work and great talk
Great presentation, very interesting topics covered
Great talk!
this is an amazing talk
Fantastic talk! But, why is _iso_ being offered to Swift only? Why not to Rust as well? ☺
Great Talk! I have one question: At 18:27, when the quorum_read is introduced: It looks like the quorum_read is for a single player only. So if you imagine that every player will play a game every Δt against another player, then the ranking might take more games into consideration for the players that are processed later in the loop in comparison to players earlier in the loop, which might give an advantage, resulting in blood.
Or alternatively, the ranking might include the win of one player over another, but not count the loss for the other player from the same game - or the other way around.
Are my assumptions correct, or does the system take that into account and the quorum_read is "lifted" to the for loop? Or are there other commands in the language that could solve this?
Hi! Good eye! The code overall is still running in a transaction block---which means it still will "appear" to be atomic, at least at that particular replica. Intersecting the per-replica transaction guarantees with the cross-replica quorum_read guarantees will result in every quorum_read reading from the same system-wide snapshot! So it works.
@@MaePiersonMilano Ah! That makes a lot of sense. Thank you for elaborating! Could have probably figured that from the word "transaction" at the top. 😅 Oh well.
Very nice talk!
Where can I find links to the three batch of papers?
This is quite cool. I just built and released a Python package that has 12 CRDTs (with hundreds of unit and e2e tests). Using them to build coordinator-less replicated storage systems is on my list of things to do by the end of the year. I'm thinking that the sha256 of a blob is its content ID, and deletions and restorations can be tracked in an Observed-Removed Set. I'm also thinking that an ORSet for insertions and deletions with each row in a sqlite table being its own Last-Writer-Wins Register (UUID4s for row ID in that case) would work pretty well.
Great and interesting talk! Sounds like ”iso” is somewhat the opposite to ”volatile” keyword. Not sure I understood it fully, but sounds useful.
Thank you for the awesome talk! I’m taking lots of ideas for a distributed systems lecture with me (mainly things I should do better)
How do I find that Derecho thing?
Isn’t this what consensus in blockchain look like ?
I thought something similar. AFAICT the talk given here is interesting as using an aspect-oriented language artifact (the strong/weak keywords) to help compile-time checking against the necessity for linearization on generic distributed programming. Consensus in blockchains would be one example of a distributed linearization algorithm that is meant to enforce the strong consistency.
But general distributed programming might have different requirements per use case on whether or not this strong consistency is required. For example, if a certain player's data always is sharded to always live on one database instance, then when you make calls for that player's personal data you don't care about what the other fresher/staler db instances might contain (weak/eventual consistency). But when you make calls that are global across all instances, then you would probably want to look at all instances (strong consistency).
That's how I'm seeing it anyway. Looks like an interesting language feature for the distributed programming universe. With blockchains specifically, maybe it could apply to cross-chain programming models.
Blockchains also need Byzantine fault tolerance. A lot of database tech assumes nodes are not malicious
doesn’t the blockchain provide far too strong guarantees so it adds unnecessary overhead?
Blockchains (to my extremely limited knowledge) invalidate data that clients initially assumed to be accurate during the recovery from a split (or whatever the blockchain people call it). This leaves a lot of work to the clients, who still need to deal with data that might not be properly synchronised yet. E.g. you might invite players under the assumption that your version of the blockchain will remain valid only to later discover that you should have invited different ones when a split gets resolved. Just waiting until enough blocks have been added for a split to be discovered being exceedingly unlikely isn't an option in many applications because it takes too long and simply synchronising all shards would be faster.