I would just like to make a small correction. I was alerted to the fact League of Legends clients don't actually have full knowledge. Starcraft would have been a better example to call out.
Heroes of the Storm is an example in the same genre as LoL where the client has full knowledge. Though that's using the same engine as Starcraft 2 I believe so pretty much the same thing. It did make for an awful experience trying to reconnect as the server wouldn't send you the latest state but replay everything from the point you were disconnected until you caught up, which could take a lot of time. As an upside you had full replay files of every match if you wanted to replay it for any reason.
Listening to some interviews with frost giant who are the sc creators making a new game, they had a very similar talk about rollback and deterministic simulation
Loved this session, it is a mind opener ! The concepts behind SpacetimeDB are brilliant! I wonder if leveraging the haxe language/vm for stored procedures would be a good fit for implementing both client-side prediction and server-authoritative computations in one common library, as you would do with webassembly... It could payoff since haxe has a rich ecosystem for gamedev (through Godot/Unity/UE integrations). Any thoughts?
Factorio is a better example. Every client runs the simulation for the entire factory. Path finding, map generation, item locations, power consumption... Connecting to the server involves, saving the game, then streaming the saved game and player updates, then running the simulation until you have caught up.
Tyler, Did you take any inspiration from Datomic/Clojure? I'm curious whether you use structural sharing to save space, and whether it's even possible to delete old, unwanted history to save space?
@@dukereg I wasn't originally aware of it when we started out. Several employees brought it to my attention though so I'm now more familiar with it. We don't using structural sharing in the write ahead log currently, although we may use content addressing to deduplicate larger column values, like binary blobs. The snapshotting system will use structural sharing in that database pages which have not changed since the last snapshot will just be referenced by their content address (like files or trees in Git). Once snapshotting is implemented in a few months you will be able to truncate the log prior to a snapshot, although data storage is cheap and as a data science engineer deleting data pains me!
These questions showed how deep the fundamental disconnect between classical programs and game development is. Instead of a transactional view (which also occurs a lot!), the gameplay code is much more a real time systems domain, and you manage very complex world and applications state dozens to hundreds of times a second.
I think it is how he was explaining it. If I tried to tell you about a horse by saying "it has legs, and you sit on it, it is brown" instead of calling it an animal, that you ride, I would need to give a lot more details before you knew I was talking about a living thing and not a chair. If he had said they were using diffing to reduce the data size just like github does, it would be real easy.
My brain is still languishing in the world of relational DB's and LAMP-stack browser games. This discussion has really yanked me into the 2020's. I experienced a lot of WOW! moments while watching this.
Thank you for making the effort to produce this video. I’ve worked both in “industrial” and “game” software engineering and I’ve always wished I could mix the two. The practice of actually writing tests for your software + the data and state centric design approach 😅. You have a new subscriber !
i cannot even fathom the amount of data that is being plumbed into a database considering its not just individuals' game data but also their controller events
wow, SpacetimeDB has some incredible potential.. and I would say as more than "just" syncing game data in real time. I'm definitely going to play with this one!
I wish more games would use the data-centric publish-subscribe pattern from DDS. Any agent should be able to subscribe to any data without much hassle.
kdb+ (Kx Systems) also can uses the database as the center of the deployed environment. q (or k) code is run inside the database and can be updated without taking the server down. For OLTP or streaming queries, the database state is kept in RAM and updates can be written to a log for replay in the event of a system interruption.
24:15 something important is that sometimes you just don't *want* the client to have all the info, even if you could. If the client has the data, then the player has the data, so you have to worry about players potentially extracting that data from memory and cheating. For example in a battle royale set in a map of several square km, if your client knows all the enemy positions the player could have some cheating software that will show that as an overlay or in some other way, giving them an unfair advantage. So if the client can show an accurate representation of the game world without that data it's best to just not send it. Of course it's not that easy because you also want the client to be able to predict future state to some degree so you cannot do things like "don't send the enemy position and velocity if it's behind a wall and not visible". Update: Ok he talks about it at 47:00 Amazing interview, guys!
If it's your first game, don't, start with an single player game. Both the tech is simpler and it's still will be playable even if it is just a few that plays it. Coding an MMORPG is hard and you also need the marketing budget to get an massive amount of players.
I think the architecture will be quite difficult to do model updates on. Because each server node and client node needs to use the same code/model. I can see a full shutdown of all clients required. Then let the server nodes sync up. Validate this. Then do model/code updates in single user mode. Link up again and validate shared state. Then update clients and re-connect them. *As for storing all historical transactions* Model changes and game rule changes mess with historical data. Historical transactions cannot really be replayed fully after a change, if it are actions (that would require the code as it was back then, bugs included). But if it is just a log of what was decided and approved, then information is lost and it becomes increasingly harder to interpret the data due to changes over time. I kind of look at it in the same way I look at persistent memory (computers that never boot up, programs that never shut down). That I will never trust, every bug is permanently ingrained in the OS/program state and permanent. For the same reasons object databases that simply act as virtual persistent memory are not trustworthy. Overal, I think it is an interesting idea, but fear that in practice it will be too restrictive and clog up over time.
this is one of the few "games are simpler" things can actually apply, and why games like MMOs have distinct servers, in a trading/4X app every user "needs" to know everything at all times, but down time can happen, so yes to update every client to the current state of the DB/Leger you would need to take everything down. in an MMO by being in say Fairy server, my client only cares about what has transpired on the Fairy server it doesn't need to know about the current state of Dragon server. the historic data is for the purpose of roll-back, so the "replays" don't absolutely need to happen, but more as an indication that the event happened, and for the purpose of the Historic Data, there would still be a separate Historic data being the World/Game State, where this would be snapshotted, and where you are taking transactional Historic Data, you can update the World/Game State Data at a slower rate; So you end up thinking of these Transactional Historic Data as "with respect too" a given World/Game State Data, so Whenever there is one of these World/Game State Data saves, that transactional data doesn't particularly matter but as a frame of reference. For the purpose of replaying it can be an important distinction, and sometimes it would be important to include deprecated functionality to maintain replay (if transaction before X-Stamp use branch Y), or it could be important to deprecate the transaction Data along with deprecating the functionality (this could cause arguments of "fairness") in a trading app, the transactions that happen "after hours" are always operated on the state of the market when it closed, and then they are reconciled when the market opens, then the transactions while the market is open are managed in real time. Then there is another snap-shot at the close of the market; at the point of that snap-shot the transactions are for historical purposes, if a say a company was de-listed at the close of trading any trades on that company do not matter, or if someone was able to inject "Historical" transactions into the system after the market closed (a bug in your situation) then that data would not be applied.
Sounds like this streams MQTT subscriptions and periodically updates the location "cell data" (I presume a few hundred meters hexagon - or overlapping squares), so one cell has the same state for every user in the cell and only the changes are streamed (UDP?), presumably in a compressed binary stream. Predictions of changes can be run on the client's state and corrected within the server tick - which could even be dynamic depending on the number of events. For persistence I guess you can snapshot parts of the server memory and write the snapshots to disk in a db like clickhouse, spark, hive etc.
At 13:00 - I might honestly be naive to the problem domain as I don't have insight directly into the code or the performance, but this seems like the approach would be a normal "piece table"? Where edits to the world is what is going across the wire and then at some interval serialize the edits. I don't see what is so hard about it. The pattern (piece table) is decades old. The only problem is making it as efficient as you want or need it to be.
Is this the decade event sourcing hits mainstream? So many people are discovering or re-discovering it. People want to swap out read models, they want historical data, and appending to a log (append only file) is basically a foot note in terms of cost.
I could see it. It feels like we finally have all/most of the tools at our disposal now. Both in terms of cheap disk and the software infrastructure to make the most of it.
The question is which aspect of software are on the top of the concerns. As in deed games are in general good at multi users, data flow, responsiveness. BUT often they lack in safety and cheating. Industry and bunking software this is priority over all and the constant service accessibility. As games should be also resistant to that in reality often it's not a case. It's not the end of the world if someone breaks your stats by cheating. But if loose savings of your life from your account, totally different story. Further more game developers agree to some level of vulnerability to be able to develop more responsive game in acceptable time and budget. So it's a often case of tradeoffs.
The challenge here is physics based sandboxes. At some level all of the example are really turn-based games that follow rules engines. Real time physics sandboxes make this stuff much harder.
Implementing SpaceTimeDB in a relational database seems completely insane to me. In order to grok the 'genius' behind that implementation strategy I'd need a VERY rational explanation of why a graph database would not be significantly preferable.
For a multiplayer game, I would send player and bullets/bomb position to both server and closest peer players to avoid server round trip for very close actions.
It surprises me how much innovation I have been seeing in the db domain recently. Just in this podcast I guess I heard also of TigerBeetle and DuckDb (from which I am a huge fan). Maybe you should interview Glauber from Turso.
Why not using a Graph database and vectors ? SQL on the mapreduce computing framework is not ideal for fast transaction on the model required for games
As an enterprisey RDBMS guy (OLAP/OLTP) who learned about ECS for the first time in context of hacking on an RTS game, what this guy is saying is music to my ears. Yes, everything is and should be a database, and if you think your application is not really a database, this just reveals your ignorance about databases.
Very interesting podcast. I my mind I see this as a continuous merge engine. Actor events coming in - Merge that with the current environment state - Stream new environment filtered on rendering distance to subscribers. So you actually subscribe to the rendering distance. All objects in the world are actors. A house that blows up is an actor like an enemy is an actor, that just runs in a pod, server side. What I am wondering about is this predictive element. I am wondering if this will be done client side. Is it possible to have the client send to most probable action based on his current state (Like where is his gun pointing or is his gun loaded), along with what he just did, so that you can merge that to send the most probable future state to all subscribers?
I don't know if i missed the question, but why on earth would u use WASM for this 🤨 feels like a weird decision, but I hope theres a good reason for it because I'm kinda hyped for this DB.
@@delibellus WASM would not be my first choice for this. I have seen that requirement pop up before, and most of the time C is used as the lowest common denominator, because like... nearly all languages have a C ABI?
@@TheHubra The main reasons here are sandboxing and transactionality. In SpacetimeDB we wrap all of the server side RPCs (we call them reducers) in transactions, which means during a transaction the only things you should be able to do are: read the arguments for the reducer, read from database tables and write to database tables. We don't allow the guest application to do any other unix-y system calls like opening files or sockets. Transactionality within reducers has a lot of benefits especially in a game development context. We also just store the wasm program itself in the database so we can always go back and replay a database regardless of the host architecture. Not that you *couldn't* do these things with just a C ABI, but WASM is generally more portable and its well supported in rust which is the primary language we used to write SpacetimeDB. With a C ABI we probably would have had to containerize the guest application which has other implications or we would have probably had to write a custom kernel which, as fun as that would be 😅, would be a lot of work. Disclaimer: I'm an engineer working on SpacetimeDB
Mentioned at 16:17 and at 53:40. It seems to me that WASM was used simply as lcd for client-side state replication that could be pushed to client at runtime. Which make sense to me since in MMO there will be an case where when you update your server state logic, there's a small window where some client might have outdated version which resolves game states differently against up-to-date client and authoritative server. Compared to C, WASM have some advantages in handling deployment while still maintaining some control to core functionality, due to their design as cleint-embedded environment. Not so different from common practice of embedding script interpreter (Lua, JS, etc) in a game to provide limited modding functionality vs exposing your engine internals directly (e.g C# detours).
27:47 as long as the server still validates every client input as according to the game rules, this is ok. But there is no way around the server needing to have game knowledge, can't trust clients to send valid data according to game rules!
How does it compare to SpatialOS and why it will perform better? How much will it cost to run this (fixed costs) and to scale up (per player hour)? What are the latency guarantees (excluding network RTT)?
SpacetimeDB is in some sense the opposite of SpatialOS. SpatialOS focused on coordinating many servers to update non-persistent state. SpacetimeDB focuses on executing as many transactions as possible on a single machine to update persistent state. BitCraft currently requires a layer on top of SpacetimeDB for coordination, but most games 99% have less than 1000 CCU in the same world, so BitCraft is a degenerate case. We do expect to build those features into SpacetimeDB eventually though. The pricing model is closer to a standard cloud service model. We charge based on the CPU and data you use, not per player, so the cost per player depends on the complexity of the game. With respect to RTT, stay tuned for our performance blog posts! We'll be posting benchmark data and our SLA for the production cloud deployment as well :)
Runescape ticks at 600ms intervals. But there are some exceptions to this. I'm wondering about security implications of this. How easy is it for my client to subscribe to things I'm not supposed to know? For example does the game have pvp areas? Can I subscribe to certain player movements that i'm not really supposed to know (yet)? Can I subscribe to the locations of all the bombs in my area? What's stopping me from doing this?
We are adding row level permissions at some point. So you as the game writer will be able to write a filter function which can access arbitrary database state to determine if each subscriber should be able to see a give row when it changes. There are a lot of important details about making this performant, but that's the gist of the idea. A Rust psuedocode example (no idea if this will show up nicely in CZcams comments): #[spacetimedb(filter, PlayerPosition)] fn filter_player_position(ctx: Context, other_player_pos: PlayerPosition) -> bool { let my_player_pos = PlayerPosition::filter_by_id(ctx.subscriber_id); // I should always be able to read my own player's position (in this case this is unnecessary because of the below condition) if my_player_pos.id == other_player_pos.id { return true; } // I can read another player's position if they are close enough if distance(my_player_pos, other_player_pos) < 5 { return true; } // By default I cannot read another player's position return false; }
@@tylercloutier537 If you're working on a game service, shouldn't this be a core principle of your design? Currently do all players get updates when another player moves regardless of their proximity? Until you implement these filter options. How is it going to get the player's current position to compare for the filter and know to only provide some of the info?
@@ferinzz These filters are already a core part of database design. This kind of code would be running server-side to filter the client's queries, so it would be using the server's recorded player position from the server database to decide access, which (should) make it difficult for bad actors to spoof with just client side access because there is no client side input beyond the request 'I think I'm here, please check I can move to this new position & give me an update on related data'. Even with a hacked client doing whatever they'd still just be calling a server-side function with their own permissions regardless of what they are doing on the client. In theory this would limit hacks to player inputs rather than wallhacks, which you need to watch for botting anyway. Because of better data history, any cheated client that left a pattern should be able to be spotted in player's history, even after they stopped using it - which would make possible aimbots or team-vision-sharing a lot riskier to use. With that history whenever you find a dodgy filter, you can go back and see who exploited it too, making enforcement a lot simpler. Its great because its all just database validation tools being applied to games, rather than a game-specific solution, so all the improvements to that system's security are available.
@@fastestdrawso wait, there's supposed to be a server that's keeping track of player data? That's not what he described. And not what the design docs describe. In the docs it's supposed to be directly from the client module to the dB module. Client tells the module that it's moving, the module might have some additional logic, but ultimately doesn't know who this player is and must at every request lookup their row of data then something will need to determine if it's allowed to move, then it updates the db. If in your description there is already a server that knows this position in between the client and the dB... what's the point of checking the db? From what I could tell the whole purpose of this development is to remove the server and have all logic be based on queries to the dB.
@@ferinzz I probably shouldn't have used the word server, sorry. Habit. The DB is running the filter, and the filters check the DB's own values, not values provided by the client exept when the the client calls functions to update those values, where the data provided goes through a validation step. The rust quickstart shows how these kinds of filters work, and you can mess with it to see how it all comes together fairly quickly. In theory plug god-knows-what into the client and still only have access to the same information and actions that you would have otherwise based on your identity token.
minecraft often had block lag.. would bit craft really be much different with it's lag with one giant server of people constantly requesting off of the database? really sounds better to have several small databases all connected changing one database doesn't really effect another. So like a per chunk basis the character can push around stuff in their chunk write stuff to the chunk they are in. Players don't have to load more than the databases required. So I guess there could be one giant game database that relies updating on the small chunk databases that change making one simple database save file for the game state but I'd say when it wants to go through 'server maintance'/turning off the database and rebooting from the save state you be sure to update the game state database with the chunk databases first. Yeah I'm kinda just overthinking how I would set stuff up.
It would have been easier to explain from the point of diffing. The Server is storing a state and the change that was made to get to that state. Just like git this allows you to roll back state without storing huge amounts of data. The Servers also use sharding for players and only send the "diff" of nearby state changes once the current state is fully loaded/cached.
Git internally doesn't store diffs, only the files themselves as blobs. Similar files are grouped into packfiles and compressed. But I agree I could have explained as diffs!
Very interesting to listen this dev talk!! just wondering dose he break the NDA by talking about the core game mechanism in the beginning? hope he wouldn't .
Coming from Unreal this sounds really weird because the server does a lot of things for you. Let's say I'm a cheater: On the client I'm gonna disable all collisions for a walls in a game. If I would walk through the wall on the client the server would correct me and teleport me back. Because on the server the collision is enabled. And the server has authority. How would I do something like this on SpaceTimeDB, how are collisions/physics handled? Because the server needs to know what the collision mesh of the level looks like. The way physics are simulated need to match how the physics are simulated on Unreal Engine. Something as simple* as movement seems like it would already be a huge undertaking.
You would need to simulate collisions on the server in the SpacetimeDB module. We already have people doing this, but it's quite a nascent aspect of the platform, that is true.
You have to understand the context of unreal, and what was said is pretty general, regardless of the engine: Unreal's physics simulation is non-deterministic: you have a variable physics simulation rate, with an experimental option to force fixed update rate. What that means is that simulation on two separate clients start to diverge over time, and you have to sync the positions and velocities of dynamic objects. Unreal replication rate is also variable - you have 2 types of replicated events - reliable and unreliable. Reliable events are strictly ordered and checked for arrival. Unreliable events can be received out-of-order or can not be received at all. Epic's approach to replication is just to spam with unreliable events to constantly update data: positions, velocities, some animation flags, etc. And send reliable events for important rare state changes. This is what unreal gives out of the box, a very general and reduntant approach, there is no saved data across time, only actual state of the world. If you want a specialized replication model - such as the one as described here, you would have to do it yourself. The implementation of reconciliation and re-ordering becomes yours responsibility. Also unreal's 2000's-java-style object model doesn't help here, because it abstracts on the object level, and not on a system's (system as in ECS), a purer/simpler data representation helps a lot, look at how Overwatch implements netcode (GDC take could be found on YT). There are examples of such approaches being implemented in UE (ex. Valorant), and we can imagine mock Counter-Strike inspired game: you have to create a separate physics world (the regular one can't be used, because we can't control how it ticks), then after each simulation you save a snapshot of the state into a circullar buffer. All your inputs are also saved and have a time assigned to them. You sink time between a client and a server. So how does a simulation look like on a server - inputs are received from client and buffered, after some delay to avoid constantly recomputing the state (the delay should be decided by a highest ping, with a ceiling of 80-120 ms) we apply those inputs and simulate the world, if new inputs are received, we splice them into the array of inputs (or if ping is too high, for ex. 200, we just discard the input, and force client to reconcile), scrap previous results of a simulation, and recompute this and consecutive updates.
@@tylercloutier537I'll be honest, that sounds like a dealbreaker. Rewriting core aspects of a game engine like physics to accommodate the server is not my idea of developer friendly.
That is by no means our primary motivation. We're just out here to build a fun game. MachineZone may have been after data mining, but having historical data is useful for customer service, time lapse videos, bug reproduction, economy balancing (notoriously tricky in MMOs), etc.
so i'm not the only coder with 'christmas tree ball's metallic purple' nail polish! 🎉 the venn diagram of coder & metallic purple nail polish must have a very small cross-section...
interesting, i can see this as a good approach for games that have complex hierarchical data like rpgs, but for simulation type games like bf/cod for example i think it's best to have everything sent to client and using strategies "delta views" (i think thats what its called) - which is just using extra compute to only send differences between states instead of duplicate states each frame to reduce bandwidth...there is a good talk on this by some engineer building battlefield/cod multiplayer server but forgot the name of the video...
also interested in trying out gundb to see if its actually possible to have a fully distributed state and offload bandwidth and latency in p2p fashion but that thing just sounds like some dark magic.
regarding the several million dollars stuff worth bought in the game, I'd like to point people to the great James Stephanie Sterling and the jimquisition (czcams.com/video/7S-DGTBZU14/video.html). No not everybody who spends an exorbitant amount of money actually has the funds, gambling addiction is a big problem, and un-surprinsingly the video game industry is using the same terms and "exploits" as the gambling industry, but due to loopholes in lawmaking video games are much less regulated than gambling sites.
Sadly, this is certainly the case as well. Machine Zone to my knowledge never had any loot boxes or other randomized sale packs, but I don't think it was necessarily because they had a moral aversion. Nevertheless it is a serious issue and one that we're mindful of at Clockwork Labs as we design the monetization for BitCraft. At the end of the day we just want to make a game which is so fun people are happy to pay for it, not exploit people.
21:38 "everything is a database" Depends on how you mean it, if that is true or not. If it means all state can be persistent, then if it not true. But if it means, every state can be accessed as if it were a database, sure, there is something to be said about that. Treating state itself as something that is persistent is enormously rigid and error prone. Every now and then someone or some institution thinks they found the light and memory needs to be persistent. That might work out well for biological systems that are inaccurate and forgetful and have Neuroplasticity. But it is a terrible concept for anything else. Think of error accumulation, having bugs forever persistent in ones program state is just plain wrong. What if a developer wants to use a different algorithm, how to transition the current state to that new one? You can see the nightmare and potential for bugs developing right in front of your eyes now, don't you?
it's a sickness that everything looks like related table structures, when you do a lot of relational db. most stuff is actually tree data structures, which are simulated on relational db, and performance suffers compared to native tree implementation. banking and gaming do indeed need fast transactions but, once in an relational db, realtime performance is not required, to say do data analysis in sql. which brings me to last point, relational db is capable of doing great big joins and filters on indexed data, which is all that sql does in the end..and that's not typically the result data you need in massive online worlds. so now that you're stuck in a relational engine for everything, you'll overcome most problems..that you've created. very similar to java (bad) developers making whole thing into objects, and then having to solve all those attendant problems, which werent original problems. i'm sure you're a capable developer but..dont fall for the falacy that everything looks like a nail, because you got a hammer in hand.
@@clray123 : so, thanks for being a dick, as you have no idea about my education and decades of working with most rdbms, oracle and teradata and embedded. you've said exactly nothing, apart that you're a dick.
That was a weird talk. I feel like his need to log historical data of everything is unnecessary for most of what you look at as a dev. It's solving retention issues without actually looking at the gameplay. Usually retention is due to failure in your gameplay's fun. If you try and balance your game around data then you're just building a Skinner box. I guess the 'run a local server client side' can make sense. But you'd need to be very selective about what can be run and determined client-side. There's plenty of instances where games made too many things deterministic based on client-side events. If it is purely to predict the next few packets, then it seems over engineered for most cases? A fireball that is going in a straight line will continue in a straight line. I dunno. It felt like he was trying to sell his DB more than talk about games as a whole after experiencing a mobile whale hunter and trying to make his own Minecraft MMO. There's I three different common setups each with their pros and cons. The Starcraft2 one where the server simply handles communication between the two clients to allow them to update each other's client info without exposing IP addresses. MMO servers which handle all of the claculations and AI on their end, then only expose as much as is necessary for the client to update their cooldowns, buffs, draw enemies/allies on screen. Peer-to-peer where usually one person hosts the game and all other players join. All the game logic is run on one player and distributed to the other players. Other players send their input events to the host who manages the logic of the input. It's the closest to playing split screen. But instead of the screen being split the events are replicated on the peer's game clients and rendered on their screen. Notice how the server doesn't rely on the 'database' to handle events and send data? Usually the database is separate from the server's loop because what's happening in real time does not need to be recorded. You don't need to read/write the position of a fireball as it travels through the air. But you do need to know when someone plopped down a tent so that other players know where it's at. A server is not a streaming database... How do you manage calculations and mutations to how skills should function. Yes you need to lookup the player stats. That's done when they log in and then held in memory. Updated when their gear or statuses change. But then processing that data is part of the server function... Which is volatile data which doesn't NEED to be recorded to a database unless you want to bloat your hard drives. At best that goes into a log file. The stuff updating the database is the current player health ONLY in case you have a game where they could log out of an instance and you want them to log in with the same health pool in the same place. There are interesting questions about how to structure your database based on what information needs to always be loaded, who should have that information at what time and how it should be accessed. Otherwise you end up with issues like in D4 where they load the ENTIRE player's stash and send that to every user who sees them. How would you manage lifetime of poison stacks on an entity? Every single poison stack has its own duration with its own damage and ticks down rounded to the nearest server tick... Are you really writing all of that to a database at every instance or are you holding that in ram and managing that stack? I'm still a beginner but the 'trust me bro it works fine' seems very fishy. Then equating it to an OS? Yeah, in a broad sense everything is a database because everything is reliant on accessing memory but... accessing memory is not the same as using it to determine gameplay events. So yeah, there's a lot of data being passed between a gamer and the server. But there's not always a ton of DB updates unless you make a DB that tracks every single event. For some reason. How much is it going to cost you to run a system like that for 1M players for 10 years? A bit more digging : BitCraft: The New Frontier in Blockchain Gaming Ah. I see. That's why he liked smart contracts. Please let me know if I missed the mark. Because with what I understand of gaming infrastructure it seems very very odd. Like trying to make a blockchain server work for real-time gaming.
6:38 the consequences of falling behind in a capitalist economy. No wonder, more people turn to online and fps games it's how they cope, inadvertently this widens the wealth gap even further.
"Everything is a database". Data is defined as the lack of uniformity in the perceived/perceivable. So if the universe was all the same thing there would be no data. But any difference is data. Everything is data, dedomena is the word. But data is not information, although it can carry it.
Hi @@Kobold666 yeah bits can represent one difference that is one unit of data which is defined as the lack of uniformity. Byte is a data structure that can be used to account for many more differences. Differences encode information but are not information. If you have a bit and only a bit you have no idea which information is encoded there if you cannot decode it. If you want to understand better the distinction I recommend reading Floriano Florido and his definition of information as semantic content. I found the phrase "everything is a database" interesting because I think everything is data, "dedomena ".
Not really. There's no cryptographic chain, and there's no decentralised consensus. The clients do attempt to predict the future state, but it's just a temporary guess. The final say rests with a centralised server. There is a similarity between deployment of the engine logic and the deployment of a smart contract, though. We discuss that in the latter part of the podcast. 🙂
@@DeveloperVoices yes, many differences, but seemingly some similar goals around having a synchronous state space with well-ordered events and state transitions, and then the developer is "sandboxed" into this clean deterministic environment by writing engine logic / smart contracts. Still working my way through the video, very enjoying
@@joelalexander928 It's amazing how many things boil down to a finite state machine. It's one of those universal patterns in programming we keep rediscovering. 🙂 Glad you're enjoying it!
Yeah. I don't want to speak for Tyler, but I get the impression he learnt a lot there but is glad he's moved on the from that company's business model.
Clockwork Labs is not MachineZone. We would not do the things they did, but it was a very interesting learning experience for me as a data scientist and engineer.
I would just like to make a small correction. I was alerted to the fact League of Legends clients don't actually have full knowledge. Starcraft would have been a better example to call out.
Heroes of the Storm is an example in the same genre as LoL where the client has full knowledge. Though that's using the same engine as Starcraft 2 I believe so pretty much the same thing.
It did make for an awful experience trying to reconnect as the server wouldn't send you the latest state but replay everything from the point you were disconnected until you caught up, which could take a lot of time. As an upside you had full replay files of every match if you wanted to replay it for any reason.
Wonderful timing on this! This is exactly what I needed for a project I’m working on. Glad to be joining the community!
Listening to some interviews with frost giant who are the sc creators making a new game, they had a very similar talk about rollback and deterministic simulation
Loved this session, it is a mind opener ! The concepts behind SpacetimeDB are brilliant! I wonder if leveraging the haxe language/vm for stored procedures would be a good fit for implementing both client-side prediction and server-authoritative computations in one common library, as you would do with webassembly... It could payoff since haxe has a rich ecosystem for gamedev (through Godot/Unity/UE integrations). Any thoughts?
Factorio is a better example. Every client runs the simulation for the entire factory. Path finding, map generation, item locations, power consumption... Connecting to the server involves, saving the game, then streaming the saved game and player updates, then running the simulation until you have caught up.
Thanks for the excellent conversation! I had a lot of fun with this!
Likewise! Thanks again for joining me Tyler. 🤜🤛
Tyler, Did you take any inspiration from Datomic/Clojure? I'm curious whether you use structural sharing to save space, and whether it's even possible to delete old, unwanted history to save space?
@@dukereg I wasn't originally aware of it when we started out. Several employees brought it to my attention though so I'm now more familiar with it. We don't using structural sharing in the write ahead log currently, although we may use content addressing to deduplicate larger column values, like binary blobs. The snapshotting system will use structural sharing in that database pages which have not changed since the last snapshot will just be referenced by their content address (like files or trees in Git). Once snapshotting is implemented in a few months you will be able to truncate the log prior to a snapshot, although data storage is cheap and as a data science engineer deleting data pains me!
This was a great conversation! Thank you both for putting it together.
Wonderful talk! Also, I noticed that you have the CLRS book 3rd edition on your shelf! Very nice. :D
Wow Kris! The banking analogy was so spot on. Really appreciate how focused you are while listening and absorbing new knowledge.
I did game development over a decade ago and this episode is one of the best things I heard in over a decade
I feel like i leveled up just listening to this talk, thanks!
Thank you for the conversation. It was eye opening for a corporate DBA.
Fantastic interviewer and interviewee. This channel is an absolute diamond in the rocks. Thanks for the content!
These questions showed how deep the fundamental disconnect between classical programs and game development is. Instead of a transactional view (which also occurs a lot!), the gameplay code is much more a real time systems domain, and you manage very complex world and applications state dozens to hundreds of times a second.
I think it is how he was explaining it. If I tried to tell you about a horse by saying "it has legs, and you sit on it, it is brown" instead of calling it an animal, that you ride, I would need to give a lot more details before you knew I was talking about a living thing and not a chair. If he had said they were using diffing to reduce the data size just like github does, it would be real easy.
1:00:39 (sings) "We do what we must because we can."
You guys at SpacetimeDB are doing some real science!
For the good of everyone (except the nodes that are dead). 😁
This has honestly got to be one of the best new channels I’ve found. Lots of really interesting videos on here!
Thanks! That's really great to hear. 😊
My brain is still languishing in the world of relational DB's and LAMP-stack browser games. This discussion has really yanked me into the 2020's. I experienced a lot of WOW! moments while watching this.
Thank you for making the effort to produce this video. I’ve worked both in “industrial” and “game” software engineering and I’ve always wished I could mix the two. The practice of actually writing tests for your software + the data and state centric design approach 😅. You have a new subscriber !
Amazing interview! Very informative. Great questions asked and very thorough and clear to understand answers.
i cannot even fathom the amount of data that is being plumbed into a database considering its not just individuals' game data but also their controller events
This was a great listen! Both spacetime and Bitcraft seem very cool. 👍
found your channel recently, very good content! congrats!
Wonderfully explained by Tyler. Plenty to digest & think about...
super, I really enjoy this kind of content especially in game. Game development is hard and any effort that try to simplify it is awesome.
Oooohhh I liked this one. I really liked it!
Thanks for the insights. It's fun to see how other engineers architect solutions to different problems.
@58:00 I was literally thinking the same right before he mentioned it - it all does sound like the actor model!
wow, SpacetimeDB has some incredible potential.. and I would say as more than "just" syncing game data in real time. I'm definitely going to play with this one!
I wish more games would use the data-centric publish-subscribe pattern from DDS. Any agent should be able to subscribe to any data without much hassle.
great interview!
Second Life (launched in 2003) also has all these features, though it's not set up as a game where you can "kill" agents.
kdb+ (Kx Systems) also can uses the database as the center of the deployed environment.
q (or k) code is run inside the database and can be updated without taking the server down.
For OLTP or streaming queries, the database state is kept in RAM and updates can be written to a log for replay in the event of a system interruption.
24:15 something important is that sometimes you just don't *want* the client to have all the info, even if you could. If the client has the data, then the player has the data, so you have to worry about players potentially extracting that data from memory and cheating.
For example in a battle royale set in a map of several square km, if your client knows all the enemy positions the player could have some cheating software that will show that as an overlay or in some other way, giving them an unfair advantage. So if the client can show an accurate representation of the game world without that data it's best to just not send it. Of course it's not that easy because you also want the client to be able to predict future state to some degree so you cannot do things like "don't send the enemy position and velocity if it's behind a wall and not visible".
Update: Ok he talks about it at 47:00
Amazing interview, guys!
btw, when he says "free-to-play game", you can read "pay-to-win game"
This is great, thanks for sharing
Thank you for show notes
You’re welcome!
Amazing interview, amazing guest.
As an academic truly appreciate👏🏾
I'm planning development for a MMORPG this was really helpful!
If it's your first game, don't, start with an single player game. Both the tech is simpler and it's still will be playable even if it is just a few that plays it.
Coding an MMORPG is hard and you also need the marketing budget to get an massive amount of players.
Great Listen
I love your nail polish, it looks so good
Thanks! 🤗
I think the architecture will be quite difficult to do model updates on.
Because each server node and client node needs to use the same code/model.
I can see a full shutdown of all clients required.
Then let the server nodes sync up.
Validate this.
Then do model/code updates in single user mode.
Link up again and validate shared state.
Then update clients and re-connect them.
*As for storing all historical transactions*
Model changes and game rule changes mess with historical data.
Historical transactions cannot really be replayed fully after a change, if it are actions (that would require the code as it was back then, bugs included).
But if it is just a log of what was decided and approved, then information is lost and it becomes increasingly harder to interpret the data due to changes over time.
I kind of look at it in the same way I look at persistent memory (computers that never boot up, programs that never shut down).
That I will never trust, every bug is permanently ingrained in the OS/program state and permanent.
For the same reasons object databases that simply act as virtual persistent memory are not trustworthy.
Overal, I think it is an interesting idea, but fear that in practice it will be too restrictive and clog up over time.
this is one of the few "games are simpler" things can actually apply, and why games like MMOs have distinct servers, in a trading/4X app every user "needs" to know everything at all times, but down time can happen, so yes to update every client to the current state of the DB/Leger you would need to take everything down. in an MMO by being in say Fairy server, my client only cares about what has transpired on the Fairy server it doesn't need to know about the current state of Dragon server.
the historic data is for the purpose of roll-back, so the "replays" don't absolutely need to happen, but more as an indication that the event happened, and for the purpose of the Historic Data, there would still be a separate Historic data being the World/Game State, where this would be snapshotted, and where you are taking transactional Historic Data, you can update the World/Game State Data at a slower rate; So you end up thinking of these Transactional Historic Data as "with respect too" a given World/Game State Data, so Whenever there is one of these World/Game State Data saves, that transactional data doesn't particularly matter but as a frame of reference. For the purpose of replaying it can be an important distinction, and sometimes it would be important to include deprecated functionality to maintain replay (if transaction before X-Stamp use branch Y), or it could be important to deprecate the transaction Data along with deprecating the functionality (this could cause arguments of "fairness")
in a trading app, the transactions that happen "after hours" are always operated on the state of the market when it closed, and then they are reconciled when the market opens, then the transactions while the market is open are managed in real time. Then there is another snap-shot at the close of the market; at the point of that snap-shot the transactions are for historical purposes, if a say a company was de-listed at the close of trading any trades on that company do not matter, or if someone was able to inject "Historical" transactions into the system after the market closed (a bug in your situation) then that data would not be applied.
Wow, sounds useful for what I'm working on :o
Sounds like this streams MQTT subscriptions and periodically updates the location "cell data" (I presume a few hundred meters hexagon - or overlapping squares), so one cell has the same state for every user in the cell and only the changes are streamed (UDP?), presumably in a compressed binary stream. Predictions of changes can be run on the client's state and corrected within the server tick - which could even be dynamic depending on the number of events. For persistence I guess you can snapshot parts of the server memory and write the snapshots to disk in a db like clickhouse, spark, hive etc.
At 13:00 - I might honestly be naive to the problem domain as I don't have insight directly into the code or the performance, but this seems like the approach would be a normal "piece table"? Where edits to the world is what is going across the wire and then at some interval serialize the edits. I don't see what is so hard about it. The pattern (piece table) is decades old. The only problem is making it as efficient as you want or need it to be.
Is this the decade event sourcing hits mainstream? So many people are discovering or re-discovering it. People want to swap out read models, they want historical data, and appending to a log (append only file) is basically a foot note in terms of cost.
I could see it. It feels like we finally have all/most of the tools at our disposal now. Both in terms of cheap disk and the software infrastructure to make the most of it.
good stuff
this man is reinventing the internet
The question is which aspect of software are on the top of the concerns. As in deed games are in general good at multi users, data flow, responsiveness. BUT often they lack in safety and cheating. Industry and bunking software this is priority over all and the constant service accessibility. As games should be also resistant to that in reality often it's not a case. It's not the end of the world if someone breaks your stats by cheating. But if loose savings of your life from your account, totally different story. Further more game developers agree to some level of vulnerability to be able to develop more responsive game in acceptable time and budget. So it's a often case of tradeoffs.
It depends on what sort of game it is. With single player game it could be good if it's possible to cheat , mess around and mod.
I'd be interested in possible applications outside of gaming, things like AI and other types of products
The challenge here is physics based sandboxes. At some level all of the example are really turn-based games that follow rules engines. Real time physics sandboxes make this stuff much harder.
Implementing SpaceTimeDB in a relational database seems completely insane to me. In order to grok the 'genius' behind that implementation strategy I'd need a VERY rational explanation of why a graph database would not be significantly preferable.
For a multiplayer game, I would send player and bullets/bomb position to both server and closest peer players to avoid server round trip for very close actions.
It surprises me how much innovation I have been seeing in the db domain recently. Just in this podcast I guess I heard also of TigerBeetle and DuckDb (from which I am a huge fan). Maybe you should interview Glauber from Turso.
Why not using a Graph database and vectors ? SQL on the mapreduce computing framework is not ideal for fast transaction on the model required for games
As an enterprisey RDBMS guy (OLAP/OLTP) who learned about ECS for the first time in context of hacking on an RTS game, what this guy is saying is music to my ears. Yes, everything is and should be a database, and if you think your application is not really a database, this just reveals your ignorance about databases.
that first 10 min. I already feel the mans internal pain that only the modern world could fathom
This seems like an excellent use case for NATS Jetstream...
Very interesting podcast. I my mind I see this as a continuous merge engine. Actor events coming in - Merge that with the current environment state - Stream new environment filtered on rendering distance to subscribers. So you actually subscribe to the rendering distance. All objects in the world are actors. A house that blows up is an actor like an enemy is an actor, that just runs in a pod, server side. What I am wondering about is this predictive element. I am wondering if this will be done client side. Is it possible to have the client send to most probable action based on his current state (Like where is his gun pointing or is his gun loaded), along with what he just did, so that you can merge that to send the most probable future state to all subscribers?
I don't know if i missed the question, but why on earth would u use WASM for this 🤨 feels like a weird decision, but I hope theres a good reason for it because I'm kinda hyped for this DB.
because of its portability and the fact that it can be used with many languages on top?
@@delibellus WASM would not be my first choice for this. I have seen that requirement pop up before, and most of the time C is used as the lowest common denominator, because like... nearly all languages have a C ABI?
@@TheHubra The main reasons here are sandboxing and transactionality. In SpacetimeDB we wrap all of the server side RPCs (we call them reducers) in transactions, which means during a transaction the only things you should be able to do are: read the arguments for the reducer, read from database tables and write to database tables. We don't allow the guest application to do any other unix-y system calls like opening files or sockets. Transactionality within reducers has a lot of benefits especially in a game development context. We also just store the wasm program itself in the database so we can always go back and replay a database regardless of the host architecture.
Not that you *couldn't* do these things with just a C ABI, but WASM is generally more portable and its well supported in rust which is the primary language we used to write SpacetimeDB. With a C ABI we probably would have had to containerize the guest application which has other implications or we would have probably had to write a custom kernel which, as fun as that would be 😅, would be a lot of work.
Disclaimer: I'm an engineer working on SpacetimeDB
Mentioned at 16:17 and at 53:40. It seems to me that WASM was used simply as lcd for client-side state replication that could be pushed to client at runtime. Which make sense to me since in MMO there will be an case where when you update your server state logic, there's a small window where some client might have outdated version which resolves game states differently against up-to-date client and authoritative server.
Compared to C, WASM have some advantages in handling deployment while still maintaining some control to core functionality, due to their design as cleint-embedded environment. Not so different from common practice of embedding script interpreter (Lua, JS, etc) in a game to provide limited modding functionality vs exposing your engine internals directly (e.g C# detours).
27:47 as long as the server still validates every client input as according to the game rules, this is ok.
But there is no way around the server needing to have game knowledge, can't trust clients to send valid data according to game rules!
How does it compare to SpatialOS and why it will perform better? How much will it cost to run this (fixed costs) and to scale up (per player hour)? What are the latency guarantees (excluding network RTT)?
SpacetimeDB is in some sense the opposite of SpatialOS. SpatialOS focused on coordinating many servers to update non-persistent state. SpacetimeDB focuses on executing as many transactions as possible on a single machine to update persistent state. BitCraft currently requires a layer on top of SpacetimeDB for coordination, but most games 99% have less than 1000 CCU in the same world, so BitCraft is a degenerate case. We do expect to build those features into SpacetimeDB eventually though.
The pricing model is closer to a standard cloud service model. We charge based on the CPU and data you use, not per player, so the cost per player depends on the complexity of the game.
With respect to RTT, stay tuned for our performance blog posts! We'll be posting benchmark data and our SLA for the production cloud deployment as well :)
Runescape ticks at 600ms intervals. But there are some exceptions to this. I'm wondering about security implications of this. How easy is it for my client to subscribe to things I'm not supposed to know? For example does the game have pvp areas? Can I subscribe to certain player movements that i'm not really supposed to know (yet)? Can I subscribe to the locations of all the bombs in my area? What's stopping me from doing this?
We are adding row level permissions at some point. So you as the game writer will be able to write a filter function which can access arbitrary database state to determine if each subscriber should be able to see a give row when it changes. There are a lot of important details about making this performant, but that's the gist of the idea.
A Rust psuedocode example (no idea if this will show up nicely in CZcams comments):
#[spacetimedb(filter, PlayerPosition)]
fn filter_player_position(ctx: Context, other_player_pos: PlayerPosition) -> bool {
let my_player_pos = PlayerPosition::filter_by_id(ctx.subscriber_id);
// I should always be able to read my own player's position (in this case this is unnecessary because of the below condition)
if my_player_pos.id == other_player_pos.id {
return true;
}
// I can read another player's position if they are close enough
if distance(my_player_pos, other_player_pos) < 5 {
return true;
}
// By default I cannot read another player's position
return false;
}
@@tylercloutier537 If you're working on a game service, shouldn't this be a core principle of your design?
Currently do all players get updates when another player moves regardless of their proximity? Until you implement these filter options.
How is it going to get the player's current position to compare for the filter and know to only provide some of the info?
@@ferinzz
These filters are already a core part of database design.
This kind of code would be running server-side to filter the client's queries, so it would be using the server's recorded player position from the server database to decide access, which (should) make it difficult for bad actors to spoof with just client side access because there is no client side input beyond the request 'I think I'm here, please check I can move to this new position & give me an update on related data'. Even with a hacked client doing whatever they'd still just be calling a server-side function with their own permissions regardless of what they are doing on the client.
In theory this would limit hacks to player inputs rather than wallhacks, which you need to watch for botting anyway. Because of better data history, any cheated client that left a pattern should be able to be spotted in player's history, even after they stopped using it - which would make possible aimbots or team-vision-sharing a lot riskier to use. With that history whenever you find a dodgy filter, you can go back and see who exploited it too, making enforcement a lot simpler.
Its great because its all just database validation tools being applied to games, rather than a game-specific solution, so all the improvements to that system's security are available.
@@fastestdrawso wait, there's supposed to be a server that's keeping track of player data? That's not what he described. And not what the design docs describe.
In the docs it's supposed to be directly from the client module to the dB module. Client tells the module that it's moving, the module might have some additional logic, but ultimately doesn't know who this player is and must at every request lookup their row of data then something will need to determine if it's allowed to move, then it updates the db.
If in your description there is already a server that knows this position in between the client and the dB... what's the point of checking the db?
From what I could tell the whole purpose of this development is to remove the server and have all logic be based on queries to the dB.
@@ferinzz
I probably shouldn't have used the word server, sorry. Habit.
The DB is running the filter, and the filters check the DB's own values, not values provided by the client exept when the the client calls functions to update those values, where the data provided goes through a validation step.
The rust quickstart shows how these kinds of filters work, and you can mess with it to see how it all comes together fairly quickly.
In theory plug god-knows-what into the client and still only have access to the same information and actions that you would have otherwise based on your identity token.
minecraft often had block lag.. would bit craft really be much different with it's lag with one giant server of people constantly requesting off of the database? really sounds better to have several small databases all connected changing one database doesn't really effect another. So like a per chunk basis the character can push around stuff in their chunk write stuff to the chunk they are in. Players don't have to load more than the databases required. So I guess there could be one giant game database that relies updating on the small chunk databases that change making one simple database save file for the game state but I'd say when it wants to go through 'server maintance'/turning off the database and rebooting from the save state you be sure to update the game state database with the chunk databases first. Yeah I'm kinda just overthinking how I would set stuff up.
It would have been easier to explain from the point of diffing. The Server is storing a state and the change that was made to get to that state. Just like git this allows you to roll back state without storing huge amounts of data. The Servers also use sharding for players and only send the "diff" of nearby state changes once the current state is fully loaded/cached.
Git internally doesn't store diffs, only the files themselves as blobs. Similar files are grouped into packfiles and compressed. But I agree I could have explained as diffs!
Doesn't Clojure/Datomic (Rich Hickey) solve both the "expensive retention of historical data" problem, and the "copying of 99% inactive data" problem?
Yes, in a sense. Datomic is awesome.
If type lazy load conect db with ai???
Very interesting to listen this dev talk!! just wondering dose he break the NDA by talking about the core game mechanism in the beginning? hope he wouldn't .
Now I'm wondering if Valve could fix their CS2 sub-tick update system with SpaceTime DB lol
This would create opportunities for more multiplayer games developed by indie devs, won't it?
My dream is that one day an indie development team can make an MMO
Coming from Unreal this sounds really weird because the server does a lot of things for you.
Let's say I'm a cheater: On the client I'm gonna disable all collisions for a walls in a game. If I would walk through the wall on the client the server would correct me and teleport me back. Because on the server the collision is enabled. And the server has authority.
How would I do something like this on SpaceTimeDB, how are collisions/physics handled? Because the server needs to know what the collision mesh of the level looks like. The way physics are simulated need to match how the physics are simulated on Unreal Engine.
Something as simple* as movement seems like it would already be a huge undertaking.
You would need to simulate collisions on the server in the SpacetimeDB module. We already have people doing this, but it's quite a nascent aspect of the platform, that is true.
You have to understand the context of unreal, and what was said is pretty general, regardless of the engine:
Unreal's physics simulation is non-deterministic: you have a variable physics simulation rate, with an experimental option to force fixed update rate. What that means is that simulation on two separate clients start to diverge over time, and you have to sync the positions and velocities of dynamic objects.
Unreal replication rate is also variable - you have 2 types of replicated events - reliable and unreliable. Reliable events are strictly ordered and checked for arrival. Unreliable events can be received out-of-order or can not be received at all. Epic's approach to replication is just to spam with unreliable events to constantly update data: positions, velocities, some animation flags, etc. And send reliable events for important rare state changes.
This is what unreal gives out of the box, a very general and reduntant approach, there is no saved data across time, only actual state of the world. If you want a specialized replication model - such as the one as described here, you would have to do it yourself. The implementation of reconciliation and re-ordering becomes yours responsibility.
Also unreal's 2000's-java-style object model doesn't help here, because it abstracts on the object level, and not on a system's (system as in ECS), a purer/simpler data representation helps a lot, look at how Overwatch implements netcode (GDC take could be found on YT).
There are examples of such approaches being implemented in UE (ex. Valorant), and we can imagine mock Counter-Strike inspired game: you have to create a separate physics world (the regular one can't be used, because we can't control how it ticks), then after each simulation you save a snapshot of the state into a circullar buffer. All your inputs are also saved and have a time assigned to them. You sink time between a client and a server. So how does a simulation look like on a server - inputs are received from client and buffered, after some delay to avoid constantly recomputing the state (the delay should be decided by a highest ping, with a ceiling of 80-120 ms) we apply those inputs and simulate the world, if new inputs are received, we splice them into the array of inputs (or if ping is too high, for ex. 200, we just discard the input, and force client to reconcile), scrap previous results of a simulation, and recompute this and consecutive updates.
@@tylercloutier537I'll be honest, that sounds like a dealbreaker. Rewriting core aspects of a game engine like physics to accommodate the server is not my idea of developer friendly.
@@Zuriki09 This is pretty standard for other multiplayer platforms like Photon Quantum.
game devs are built different
when your primary motivation is data mining, but to get the data out of the plebs you also need to develop some game they will find addictive enough.
That is by no means our primary motivation. We're just out here to build a fun game. MachineZone may have been after data mining, but having historical data is useful for customer service, time lapse videos, bug reproduction, economy balancing (notoriously tricky in MMOs), etc.
Bookmark 31:44
so i'm not the only coder with 'christmas tree ball's metallic purple' nail polish!
🎉
the venn diagram of coder & metallic purple nail polish must have a very small cross-section...
Haha, together we can expand!
Listening at the moment - I'm getting the impression that they have invented QuakeWorld circa late 1990s
?
Jusf wow
interesting, i can see this as a good approach for games that have complex hierarchical data like rpgs, but for simulation type games like bf/cod for example i think it's best to have everything sent to client and using strategies "delta views" (i think thats what its called) - which is just using extra compute to only send differences between states instead of duplicate states each frame to reduce bandwidth...there is a good talk on this by some engineer building battlefield/cod multiplayer server but forgot the name of the video...
SpacetimeDB also only sends the deltas between states, depending on what you mean by delta. i.e. we only send rows to the client that have changed.
also interested in trying out gundb to see if its actually possible to have a fully distributed state and offload bandwidth and latency in p2p fashion but that thing just sounds like some dark magic.
Yeah you're not getting "automatic" prediction, You're just doing prediction on everything. I guess that's a way to do things.
at minute 4 may I know the game name? THe one Tyler say you can feel like a king?
Game of War: Fire Age
Space-time? Shout-out to Alfred Korzybski.
regarding the several million dollars stuff worth bought in the game, I'd like to point people to the great James Stephanie Sterling and the jimquisition (czcams.com/video/7S-DGTBZU14/video.html). No not everybody who spends an exorbitant amount of money actually has the funds, gambling addiction is a big problem, and un-surprinsingly the video game industry is using the same terms and "exploits" as the gambling industry, but due to loopholes in lawmaking video games are much less regulated than gambling sites.
Sadly, this is certainly the case as well. Machine Zone to my knowledge never had any loot boxes or other randomized sale packs, but I don't think it was necessarily because they had a moral aversion. Nevertheless it is a serious issue and one that we're mindful of at Clockwork Labs as we design the monetization for BitCraft. At the end of the day we just want to make a game which is so fun people are happy to pay for it, not exploit people.
Why is Kris' webcam set for such a low frame rate? Sharp images but a bit jarring.
There’s not much more to it than moving data around in memory or recreating it.
Okay, sounds like firebase. How does it scales?
21:38 "everything is a database"
Depends on how you mean it, if that is true or not.
If it means all state can be persistent, then if it not true.
But if it means, every state can be accessed as if it were a database, sure, there is something to be said about that.
Treating state itself as something that is persistent is enormously rigid and error prone.
Every now and then someone or some institution thinks they found the light and memory needs to be persistent.
That might work out well for biological systems that are inaccurate and forgetful and have Neuroplasticity.
But it is a terrible concept for anything else.
Think of error accumulation, having bugs forever persistent in ones program state is just plain wrong.
What if a developer wants to use a different algorithm, how to transition the current state to that new one?
You can see the nightmare and potential for bugs developing right in front of your eyes now, don't you?
sounds like Game of War-Fire age, or Evony
Game of War I think. :-)
@@DeveloperVoices I dropped about 30k on that game 😬
@@criticalcitizens Ouch! I hope that was all money you could afford. 😨
Take a look at Star Citizens programming problems they're trying to solve! They're doing about as advanced an architecture as I've ever seen
Distrubuted systems are complex ?
Only if your model is bad or heaven forbid code is spagetti.
Didnt warcraft 3 have rrplays 23 years ago?
Rubber banding due to ping is a bad experience, I don't see how using global database is a good idea for this MMORPG?
it's a sickness that everything looks like related table structures, when you do a lot of relational db.
most stuff is actually tree data structures, which are simulated on relational db, and performance suffers compared to native tree implementation.
banking and gaming do indeed need fast transactions but, once in an relational db, realtime performance is not required, to say do data analysis in sql.
which brings me to last point, relational db is capable of doing great big joins and filters on indexed data, which is all that sql does in the end..and that's not typically the result data you need in massive online worlds.
so now that you're stuck in a relational engine for everything, you'll overcome most problems..that you've created.
very similar to java (bad) developers making whole thing into objects, and then having to solve all those attendant problems, which werent original problems.
i'm sure you're a capable developer but..dont fall for the falacy that everything looks like a nail, because you got a hammer in hand.
Just because all you know is MySQL does not mean that RDBMSes that can natively handle trees do not exist.
@@clray123 : so, thanks for being a dick, as you have no idea about my education and decades of working with most rdbms, oracle and teradata and embedded.
you've said exactly nothing, apart that you're a dick.
Dear lord, not stored procedures...
😂
The guy looks like an old wizard from lord of the rings lol
I absolutely despise Machine Zone
That was a weird talk. I feel like his need to log historical data of everything is unnecessary for most of what you look at as a dev. It's solving retention issues without actually looking at the gameplay. Usually retention is due to failure in your gameplay's fun. If you try and balance your game around data then you're just building a Skinner box.
I guess the 'run a local server client side' can make sense. But you'd need to be very selective about what can be run and determined client-side. There's plenty of instances where games made too many things deterministic based on client-side events. If it is purely to predict the next few packets, then it seems over engineered for most cases? A fireball that is going in a straight line will continue in a straight line.
I dunno. It felt like he was trying to sell his DB more than talk about games as a whole after experiencing a mobile whale hunter and trying to make his own Minecraft MMO.
There's I three different common setups each with their pros and cons.
The Starcraft2 one where the server simply handles communication between the two clients to allow them to update each other's client info without exposing IP addresses.
MMO servers which handle all of the claculations and AI on their end, then only expose as much as is necessary for the client to update their cooldowns, buffs, draw enemies/allies on screen.
Peer-to-peer where usually one person hosts the game and all other players join. All the game logic is run on one player and distributed to the other players. Other players send their input events to the host who manages the logic of the input. It's the closest to playing split screen. But instead of the screen being split the events are replicated on the peer's game clients and rendered on their screen.
Notice how the server doesn't rely on the 'database' to handle events and send data?
Usually the database is separate from the server's loop because what's happening in real time does not need to be recorded. You don't need to read/write the position of a fireball as it travels through the air. But you do need to know when someone plopped down a tent so that other players know where it's at.
A server is not a streaming database...
How do you manage calculations and mutations to how skills should function. Yes you need to lookup the player stats. That's done when they log in and then held in memory. Updated when their gear or statuses change. But then processing that data is part of the server function... Which is volatile data which doesn't NEED to be recorded to a database unless you want to bloat your hard drives. At best that goes into a log file. The stuff updating the database is the current player health ONLY in case you have a game where they could log out of an instance and you want them to log in with the same health pool in the same place.
There are interesting questions about how to structure your database based on what information needs to always be loaded, who should have that information at what time and how it should be accessed. Otherwise you end up with issues like in D4 where they load the ENTIRE player's stash and send that to every user who sees them.
How would you manage lifetime of poison stacks on an entity? Every single poison stack has its own duration with its own damage and ticks down rounded to the nearest server tick... Are you really writing all of that to a database at every instance or are you holding that in ram and managing that stack?
I'm still a beginner but the 'trust me bro it works fine' seems very fishy. Then equating it to an OS?
Yeah, in a broad sense everything is a database because everything is reliant on accessing memory but... accessing memory is not the same as using it to determine gameplay events.
So yeah, there's a lot of data being passed between a gamer and the server. But there's not always a ton of DB updates unless you make a DB that tracks every single event. For some reason. How much is it going to cost you to run a system like that for 1M players for 10 years?
A bit more digging : BitCraft: The New Frontier in Blockchain Gaming
Ah. I see. That's why he liked smart contracts.
Please let me know if I missed the mark. Because with what I understand of gaming infrastructure it seems very very odd. Like trying to make a blockchain server work for real-time gaming.
"speed ups" aka pay to win
Can openAI help???
No.
6:38 the consequences of falling behind in a capitalist economy. No wonder, more people turn to online and fps games it's how they cope, inadvertently this widens the wealth gap even further.
4X: Explore, Expand, Exploit ... EXTERMINATE... EXTERMINATE... EXTERMINATE...
240hz ...
Why not us surrelql
Ooh, maybe we should do an episode on SurrealDB. 🤔👍
Runescape updates every 0.6 seconds.
Yes, you're right, I misremembered this.
"Everything is a database". Data is defined as the lack of uniformity in the perceived/perceivable. So if the universe was all the same thing there would be no data. But any difference is data. Everything is data, dedomena is the word. But data is not information, although it can carry it.
But we're talking about bits and bytes. Bytes are data and they encode information because there are no empty bytes.
Hi @@Kobold666 yeah bits can represent one difference that is one unit of data which is defined as the lack of uniformity. Byte is a data structure that can be used to account for many more differences. Differences encode information but are not information. If you have a bit and only a bit you have no idea which information is encoded there if you cannot decode it. If you want to understand better the distinction I recommend reading Floriano Florido and his definition of information as semantic content. I found the phrase "everything is a database" interesting because I think everything is data, "dedomena ".
isn't he basically describing blockchains?
Not really. There's no cryptographic chain, and there's no decentralised consensus. The clients do attempt to predict the future state, but it's just a temporary guess. The final say rests with a centralised server.
There is a similarity between deployment of the engine logic and the deployment of a smart contract, though. We discuss that in the latter part of the podcast. 🙂
@@DeveloperVoices yes, many differences, but seemingly some similar goals around having a synchronous state space with well-ordered events and state transitions, and then the developer is "sandboxed" into this clean deterministic environment by writing engine logic / smart contracts.
Still working my way through the video, very enjoying
@@joelalexander928 It's amazing how many things boil down to a finite state machine. It's one of those universal patterns in programming we keep rediscovering. 🙂
Glad you're enjoying it!
so its a blockchain xD
While very interesting I would not stand behind a company that makes such a game. I would not like to work on that for ethical reasons.
Yeah. I don't want to speak for Tyler, but I get the impression he learnt a lot there but is glad he's moved on the from that company's business model.
Clockwork Labs is not MachineZone. We would not do the things they did, but it was a very interesting learning experience for me as a data scientist and engineer.