Star Citizen: Network Engineer on Server Meshing
Vložit
- čas přidán 31. 12. 2023
- As a Network Engineer and player of games since MUDs and Doom....I thought I'd kick this channel off with my thoughts on Server Meshing and some gameplay showing how bad a pilot I am.
I can finally credit the Video that spawned this whole thing:
@free-dissociation - • A Software Engineer Re...
I have nothing else to go down here yet, perhaps some day I'll be a big-boy youtuber with links and all sorts of stuff no one reads for the bottom of my description! - Hry
"Think of this like the metaverse only not stupid" sums it up quite nicely 😀
Got a dry chuckle from me for sure 🧐
From now on, if I want to explain to someone, why SC is more than just another Space-Sim, I'll totally use this 😀.
It kinda *is* a meta verse. A simulated reality outside of our own with real world value earned and generated in it.
I don't think it's a GOOD label for it, but from a definition standpoint it's technically correct.
@@Briggsby that description could describe networking in general
"Let's look at a simple graph" - shows accurate model of the human brain
Experts giving their 2 cents on stuff I love is some of my favorite content!
Also, Happy New Year 🥳
Thank you all for your comments, I'm doing my best to read and where I can, answer questions. The outpouring of responses here is more than I could have ever expected, it's amazingly gratifying. I AM reading every comment, but I'm unlikely to be able to respond to each and every one.
Amazing content, great way of explaining and I will say I got lost a few times haha.
Like your nutral way of giving info
This was really interesting to listen to. I would definitely tune in if you were to review more of their tech, or go more in depth in the future.
I would definitely tune in to hear what you have to say on the foundation of StarCitizen.
As a fellow 30 year network engineer I'd love to see a deep dive into the architecture and routing/switching of the infrastructure.
Right there with you, have been holding my breath for years now hoping they would do a deep dive into this and give us more information about how they are doing it. Hopefully once it gets implemented and more stable they will do some sort of round table and fill us in. I think they should because that would put a lot of these back seat know it all wanna-be engineers in their place and shut them up!
@@kevinm3751 I'm just a laymen; but wouldn't this tech be best packaged for third party sale? And if they release said info you want to see, wouldn't that just tell everyone else how to make it themselves for free? Or is it not that simple?
Doubt they will reveal their infrastructure and network layer information, that seems extremely irresponsible for security purposes alone.
@@craigwoodward8455 How compatible is your code?
Another NetEng here (but principal NE nonetheless) and my opinion (and the point I think he's getting at) is that while network engineering principles can be a useful way to view the problems it's inherently a server communication problem. I very highly doubt that CIG has a ridiculously complex network architecture. The KISS rule applies, if you can simplify an area, do it. At least that's my gut feeling for how I would look at it in a professional capacity. Packets go places, and you want them to get there fast and in a hurry. It's what's done with those packets and what is included in those packets that is important.
Do more, please. Always love hearing from experts, it is so easy to dismiss something as easy when you cant grasp the work require to achive the set goals.
I've been saying for a long time that if they can pull off dynamic server meshing to a level where other game developers want to licence it from CIG then that in itself will pay for SC's server costs for the next decade (or more). I'm sure there's even the potential for this tech to be used outside of the gaming industry.
they recently stated they have no intention on selling this tech once they complete it. but im sure that will change
I mean, if they actually pull it off they've basically invented an infinitely scalable compute solution. Got a problem? Apply as many servers to solving it as you like.
It's a much bigger deal and more difficult concept than most people realize, with potential applications that range from videogames to protein folding.
More. Say more on this. I've not heard anyone with as much insight and experience in the field speak on this subject. Your piloting will improve over time so long as you continue playing. EXCELLENT WORK!
I was a CCIE back in 2014 or 2015. I've moved over into virtualization now, but can still appreciate the complexity of what they are doing. I was so hyped up about the announcement, I jumped back into SC for the first time in about a year. There was a player in the server just ramming players on their pads. Moral of the story - the players will always find a way to abuse and ruin whatever awesome stuff is implemented.
Not the player but more like the asshat will find a way to be a better asshat 😊
If you take a couple photos and report him. They will strike him with a ban hammer for a couple weeks. More if he keeps doing it. They'll let you do a lot of trolling,, but they draw a line at that stupidity luckily.
hopefully they figure out elevators and hud dimming
Landing pads haven't been around for over a year though...
@@Briggsby what? Landing pads have been in the game as long as I have been playing which is over 2 years.
"Think of this like the metaverse, only not stupid" Take my sub and my like lol
Always interesting to see a bit more deep dive from a professional. I myself just know it's unbelievable and you are the one to explain why 🙂
This was great! More of these please. Would love to hear your point of view on this topic as this progresses.
I'm definately subscribed. That was a very informative breakdown of what they are trying to do with replication and server meshing. Thank you for this!
I really enjoyed this, moving on to the next part. It is very informative and helps me understand exactly what they are working on and trying to do. Thanks for the video.
Im glad to hear someone with expertise weigh in on it. (and personally happy at the positivity you see towards the project)
Great video mate! I too would appreciate more content around the networking side of SC!
I'm so impressed with your understanding and the specific ability you have to share that information to us. THANK YOU! Backer from 2014
One hell of a first video man! Well done!
This is great Tybio! Subscribed! As a 20+ year (primarily mobile client) developer, not a networking expert, I was absolutely blown away by what they're trying to accomplish with server meshing and replication layer. As you said, this is much bigger than SC and has applications not only in other games but in the entire framework of what a meta-world could even be. There is absolutely no reason why a root node transition from, say, Stanton to Pyro, would be any different from a transition from a virtual city through the front doors of a virtual Porshe dealership in a real "bigger-than-Facebook/nVidia" sense. If they can transactionalize the handoff of a node transaction in an NFT this is a technology industry shaker not just a gaming industry shaker.
This is how the real Oasis will be built…
Love the shout out to the old Wow Gates of An-qiraj event :D
oh dang comin in HOT earning that sub with the first vid. cant wait to see how your channel develops
This was an amazing perspective and I'm happy to see someone else shed light on CIG's pioneering endeavor to make a game of this scope. It does hold massive potential of future implications for the entirety of gaming and many people just don't understand that. Thanks again!
Software developer speaking here:
Ideally the specific nodes would be grouped dynamically depending on player count. So for example if there's only 3 players on a ship the whole ship would represent a single node, whereas if 100 people would stick together on a Carrack you eventually might have the Carrack-Node broken down into nodes for the distinct rooms etc.
Or another example:
If Microtech were it's own sub-graph and each building a sub-node, then you wouldn't need to maintain all the sub-nodes in case no player being on Microtech at all. Sub-Nodes for Microtech only would be created when some player enters the Microtech parent node etc.
In this way the overhead of maintaining huge graphs can be minimized while at the same time providing good scalability.
And I totally agree: I'm also quite confident, that they will get it right (even if it might take another 1-2 years). But then they have this HUGE engine they can sell to basically any other Publisher. Eventually selling licenses to the engine could generate a much higher income then they ever could get from SC or SQ42 alone.
This was really enlightening. I’ve been a Star Citizen backer since 2016, and just recently started my freshman year at college majoring in Information Technologies & Management where we’ve started to delve into server architecture and networking. In reality this tech they’re making could fund the game immensely, it’s ground breaking really, and could change the MMO industry forever. This would effectively immortalize cloud imperium games.
this is your first video, and you said something i haven't seen someone say well. Such great context and information
Great video, blew it out of the park for a first video, looking forward to more!
It's great to get a the perspective of a networking engineer on the server meshing. SC is a crazy idea to even attempt, and CR is probably the only person who could pull it off, and I I hope so much they succeed. however as a software engineer I was a bit horrified by the server meshing demo at cit con. It looked great and seemed a logical solution, but I just don't see the scalability of the solution.
There has to be more going on in the Background thst we did not get to see or there is going to a huge scalability problem going forward
It could be they are hiding the smarts behind the covers. But I don't see any easy solution.
You are spot on, a game of this scale is crazy to attempt, and I am in awe or either their smarts, bravery or naivety to even attempt.
as another SW engineer, yes, but I do think there are solutions to those problems, maybe it will take a while, but there will. I remember when planetary tech was introduced (being more or less persistent already back then) I was like "how the hell?", especially given that originally that wasnt even part of their "framework". Now its as normal as anything else...
Yeah there was few "items" when ppl thought that it would be impossible and or rly hard to scale, and yet they did
Well said!
I just want to point out, there is a bug in your coding above.
Watching Mark fix bugs was fascinating to me.
Bear in mind that the CitCon demo was (afaik) a re-run of their internal demo that was intended to show that the technical 'underpinnings' were done, rather than show how the system would scale. Paul and his team got Server Meshing 'working' just weeks before CitCon, hence the limited nature of that demo.
Beyond that, on paper the architecture will scale (although 'how far' will be dependent on *many* factors) - but the initial implementation likely won't... at least, not significantly. CIG have always said that the intention is to get the architecture in place, and then iterate on the actual code to improve performance and scale.
Please please do not forget: SC has said alot of things over time but has difficulty delivering on what they say. Never discount that it may be oversold, inaccurate, or maybe even just entirely marketing. It's one thing to talk about something, promise it, say you're working on it, etc. It's another entirely to deliver a working solution. CIG is very good at SELLING their game. But we're still waiting 10+ years later for the real rubber to hit the road and prove that they can do what they've been saying they're gonna do for all this time. There are many promises they have broken over time, such as Hex Code Colors where they promised backers fully customizable ship coloring and customization and even stated they were using those system in house already and just needed to translate them to a end user experience. And instead years later they started selling skins and that feature has never actually made it into the game and prolly never would as it would now be a conflict of interest.
I love these kind of videos. My inner nerd enjoys it to the extent you could have talked for an hour, and i would have listened.
Great video.
Bring is more of this ! Your Input is amazing and so worthy an needed for the Community and newbies.
Many people don’t understand the complexity of this game and hearing that stuff from an expert is quiet intressting. Wish you all the best for your Channel. Hope to See more from you !
Great video i look forward to listening to more from your perspective.
"Think of this as the Metaverse, only not stupid." - it was at this moment I subscribed! :) Thanks for the thoughtful analysis - always great to get a deeper glimpse from folks with expertise in the subject matter!
Omg, agreed with your conclusions 1000%!!!
I've worked in networking and as a sysadmin for 15 years, and I extrapolated the impact their server meshing tech would have on the entire gaming industry, and maybe much more just like you did! Finally, someone that understands!!! Being able to seamlessly devide any virtual space up across many different physical and logical machines for many, many people is basically the holy-grail of game design. They're going to sell that with StarEngine to everyone and online games will start getting bigger and bigger. Population limits of servers could be immensely increased, or maybe even virtually eliminated! I have been losing my mind ever since the citizencon demo ever since. Again, I think you are exactly right in your conclusions, and yes, grouping up into large groups during the replication testing will definately be the best way we can assist them in gathering the vital data they need to make this work!
Absolutely love your message at the end, beautifully said!
"Think of this as the metaverse, only not stupid" 👍
Definitely appreciate your perspective; just retired after 39 years in IT ..server, cloud architect. Must admit you raise a good point about the nested tree structure and associated complexity. You wonder how much overhead will be taken up by pure dynamic management of it vs other player transactions. I definitely appreciate what they're trying to object/inheritance wise..but I sure hope they've got really powerful servers
as someone thats going for their CCNA this was real cool to listen too and get an insight on
Fantastic video, you have a great cadence for content
Great job dude! On also letting us know how we can help CGi figure it out :) We should plan when it comes out to gather somewhere as a test!
Thnx for your insights! You are doing good, keep them coming mr!
Love this! Thank you for covering this! You definitely earned a sub!
This is excellent material. As a website developer and nowhere near your level, I've been waiting for you. Keep going! 😊
Great video. Awesome job. Thank You kindly.
thank you so much for your input
That was really good.. You are very easy to listen to. Keep making videos. You have alot to teach, show and react to. We would love to hear more of your thoughts!
Pilot tip: Dive straight down/into hangers and face your aim to the center of the hanger until you are all the way inside. Then worry about rotating your ship to land on the deck.. See you online and keep up the great work!
Just stumbled on here! What a marvelous first video eh :D I especially liked that conclusion part, I wholeheartedly agree. I'm a game dev, I too share those feelings you described :D
And as for CIGs "mistakes", that's just part of the process mate, surely you know it too, thing is with CIG we just all get to see their mistakes, people are a little to hard on them for that; We're seeing the sausage getting made, not everyone has the stomach to watch the process
Great video! Thank you for your valuable insight. I hope everyone in the SC community has a chance to watch it to get a sense of the gravity of what is being accomplished by CIG!
I'd love to be a fly on the wall in some of CIG's engineering meetings. Greetings from an old gray beard Linux guy
As a fellow (albeit more recent network engineer) this was great content, and from a beginner's perspective is truly excellent educational material on it's own. Massive kudos.
I just subscribed to your channel Sir. You are doing something very important for the community and that is education. Thank you.
"Like the Metaverse but not stupid" 😂 I loved that. This really seems amazing.
Never stop yourself from making an analytical video like this, Sir. Thank you for your work. The passion i'm hearing is the same thing im feeling for this "thing" they are making. I sat on the fence much longer than most, but never wrote them off once. What i see now is nothing short of a technical miracle. Been in IT a LONG time.. i know what im looking at. Unbeleivable, really.
great video, an expert breaking things down. thank you
I was glad to hear you touch on simplification up the tree -- as object containers will not need to notify other object containers of every event that occurs within them. This is one of the keys of making this all possible -- determining what information can stay in the container and what information needs to be shared. With this type of system, an entire ship in space could produce as much data as a single player in a ship with respect to the objects in which both reside. Meanwhile, tons of things can be happening within each ship that are confined to each's object container.
The concern I have will be the need for distributed databases to track it all. Each object container will effectively be its own database and those updates will have to be committed back to a "master" database (at least in function) at some point. How they handle this particularly as dynamic meshing comes online will be critical.
The real complex part about this is that when it comes time to sync there is more than one possible authoritative node. This in of itself is normally a huge no no, but I can't see any way around it. It should be OK because it is game data and not something like finance etc and some corruption is acceptable. Normal distributed DBMS have a master write authoritative node and many replicated read nodes.
@@Diogenes76 Rewatch the server meshing demo. Only one node is authoritative.
Really interesting take! Always cool to hear someone who actually knows how this stuff works give their insight on things. Tonna share it with the org I'm in!
loved the video! love the game, and it's great to see a professional give a informed opinion.
Subbed, we have similar backgrounds. Mine in operations and architecture in the compute space, including cloud more recently. Sounds like we might have similar views on this. I’m interested in seeing more of this type of discussion from network, storage, compute, cloud, developers, etc. To really establish a credible source of information on the community side since there are so many misconceptions, for a variety of reasons.
I really appreciated hearing the network architecture perspective, it’s been a gap for me.
Holy hell!! Someone gets it!! Thank you so much brother, I have an IT background and I baffled by the scope of what CIG is doing.
Excellent first video, sir.
Awesome video, you 100% hit the nail on the head at every turn! Subscribed for more.
very interesting. thanks for your take on this
As a much younger network engineer, thank you. You've put this so eloquently, and accurately. I know I'm going to spread this video as widely as I can.
"think of this like the metaverse but not stupid." on point sir.
Interesting and lighted point of view
Very interesting and accessible to those of us who aren’t engineers. Thanks for sharing this.
Quality content here. Bravissimo from Italy!
thanks for the insight.
Thanks enjoyed that, please do more!
Thank you being knowledgeable about the topic you talk about in regards to this game, a lot of people do not have the will or ability to understand "what the hold up is", I feel this may help a vast majority of people understand the sheer scale of not the just game but the technology required to make it function.
I feel like we're missing some good video explaining what SM / RL is for non tech people. I work in the field as well so I THINK I have a fairly good understanding of what they're going for (and it is truly mind-blowing), but regular people don't have the faintest clue and just repeat key word they heard without understanding.
Ideas:
- Explain how a "normal" game works (= 2022 star citizen), with client DGS concept
- Explain how 3.18 with PES changed things and we moved into the concept of a shard and why
- What the RL will do (possibly details about the micro services & Gateway service )
- What/how server meshing will work
Probably explain the challenges, why each step is important, etc
Great video anyway, thanks for that !
Great vid! Thank you.
The graph facilitates a way for compute to be assigned to Nodes in the graph, and the Edges define the relationship between the Nodes to determine what is computing events. Players moving around and performing actions emits Events, and the Events are attached to Nodes. This way, the compute power assigned to each Node is responsible for processing the events emitted by players, the events emitted by NPC actions, and determining the effects and interactions that need to be transmitted back to players.
The complexity of the graph is not important, as a well designed system will be able to dynamically restructure the size of the graph to introduced more compute for nodes that are under load. E.g. If 50 players walked into a single ship, then that ship can be removed from the compute power of the node it was previously in, assigned to a new compute node, and now events assigned to that ship will be managed by a dedicated compute node. Later when everyone leaves the ship, the compute node can exit the scene, simplify the graph by removing the dedicated node for this ship, and go off to deal with compute requriements elsewhere in the graph, depending on load.
So, the graph shows the structure of compute capacity and how players are handed off between them, and events are assigned to the compute power assigned to each node.
If done correctly, this can handle a large amount of traffic, the challenge is to reduce rubberbanding between nodes, and have the capability to divide and merge nodes very quickly based on demand.
That's how I would do it, based on years of work in graph databases and event queues.
Interesting to have a network engineer point of view on this, the data exchange will indeed be huge and will be something critical. I really curious about the details of their technology, this is truly fascinating!
That this is the part of the project that hooked me. This is the Tech that excites me, I'm also in networking but deal with problems that are well understood, the ones CIG has to solve are pretty fascinating and I'm sure require experts from different disciplines working together on bespoke systems. From the outside looking in, it does sound like the bleeding edge.
LOL @ crushingly honest description!
and great thoughts, thank you
Really interesting to hear the point of view of a senior Network engineer. Your enthusiasm is refreshing.
Now, the important bit I want to share: LANDING TIP! haha. When you land vertically, put the camera on top of your ship (hold the Z key to move the camera freely) and then go into decoupled mode: now strafe down just a tiny bit to give the ship an impulse, it will continue to descend slowly. Then all you have to do is to slow your descent with the brake key (X) and maybe strafe/roll to ajust your position. Voila, have a great day!
"like the metaverse just not stupid." 🤣
dude thank you! i was hoping for a specialist sharing their oppinion.
i would love to see more "meta"-talks especially about this topic.
thx alot!!!
Super video! I think it's great that you as an expert don't say, I haven't seen that yet, it can't work. Instead you give an objective and well-founded opinion and show us ways to help CIG! Unfortunately very rare on CZcams!
Thank you for your take. This will get some more traction, and so will your channel if you can explore the tech side further.
That's the plan! Thanks for your comment!
"Think of this like the metaverse only not stupid." 🤣🤣💀 I'm literally dying after hearing that roast!!
Thanks for opening up this conversation. So many players have zero clue about how complex of a problem this is and what has to happen to get it working and even from what you just said only touches the tip of the overall scope of this functionality and the complexity of implementing it. Hats off to CIG for what they have accomplished but not sure why everyone was so excited a few weeks ago when they presented this to us because that shows they are still a long way off from realizing this as something that can be put in the game at any scale! Still it is awesome they are willing to take on this challenge and we all need to be patient and hold our breath and hope they can get it all figured out because this will be the biggest feature of the game in the end and the amount of things they can hook to it is going to be where the real cool stuff begins to happen!
Thank you very much for your professional opinion on this subject. Was really interesting to listen. Liked & subbed
About what you said at the end of the video.
This video certainly is NOT a mess.
Great content!
I did CCNA1 and 2, I also am a server administrator, and cloud environment administrator and I am a programmer and also a web developer, that also does Graphic Design and multimedia. So I have said to many that Server Meshing is the holy grail of MMOs so I agree with you. From my limited scope of dabbling into many professional IT paths over the years.
Great video, tks for sharing your view!
The network part of this game is their greatest challenge, I'm really curious to see how 2024 goes, since some of the systems are supposed to come online.
Haven't really played since 3.16, so hopefully it will be a nice surprise.
Fantastic explanation!
Hey great first video, entertaining and informative 👍
Awesome, thank you!
This was really insightful, thanks for sharing.
One thing I keep hearing from other, sceptical people is that things like server meshing have already been done in game "X, Y or Z" and therefor it's not a big deal or something. I do wonder what the difference is between other games using some kind of servermeshing and CIG's attempt. I understand it's all very technical and honestly, most of it goes way over my head but I do think CIG is actually pushing boundries. I hope they succeed.
I thought that Vulture would explode at any moment. The entrance to the hangars is very narrow. ^^ Great video. I agree.
Good stuff mate, cheers
Regarding population congregation - What I found interesting about the recent CitizenCon demo was they were actually able to split up a very small "L" shaped room into 3 separate servers and transition all of the actors seamlessly. Obviously this was just a demo of their capabilities and they probably won't use that level of granularity, but the point is THEY COULD if they wanted to. They could determine the absolute maximum number of players packed shoulder-to-shoulder that would fit into a space station, calculate the area, and then divide that space onto it's own servers and there would theoretically be NO LIMIT to the number of players that could play together because they would first reach their physical space limit before they would reach the individual server capacity limit. And with their ability to quickly tear down and spin up servers as the population moves around the physical space, this gives them HUGE flexibility when it comes to events that would bring large numbers of players together. I know the technology they have developed is tremendous and I can't wait to see what they can do with it. (P.s. I'm a Software Engineer)
The down side of that is the processing overhead that comes with it, the smaller the slices are, the larger the percentage of the work is overhead. I have no idea what that overhead is, but if you have a big room with one server than all the work happens in one process, handing that around between 4 servers that have to have complete visibility into each other will eventually cost more than it gains. I love the thinking though, it could be that simple, but I have reservations :)
@@MrTybio I think the key is whether 4 servers would need "complete" visibility or not. What the servers need to see is different than what the players need to see and CIG has indicated that the entities that each views would be different. Since the servers deal with physics, they only need to have a view at the entities they have authority over as well as any entities that might interact with their authorized entities, but that could still be separated by spacial zones even if there aren't in-game physical walls defining them. A good example of this might be the single large room in the convention centers, it may actually get divided up into multiple physics zones that individual servers could handle. Each server would only handle the authorized entities within their zones but they will likely have additional unauthorized entities that either are intersecting multiple zones (like the floor) or on a path to their zone. They wouldn't need to see any entities within a zone that doesn't share a zone border with them.
The player view would be different and would be based more on line of sight and might require more difficult handling than the physics servers because players actually might need "complete" visibility. Although CIG has mentioned that there could be techniques for reducing the load on clients by prioritizing entity updates based on how close those entities are to the player so for example the people closest to you in the convention center would update every frame but a person on the far side of the convention center might update with a lower fps but it might not be as noticeable to players because the foreground movement is more likely what the player would focus on.
Good video, rough editing but I understood your point. I'm only a level 1 tech but I have a basic understanding of networking. And yes what they are undertaking is massive attempt at a new way to do server hosting. It will be interesting. I'm not worried about the desyncing, I'm more worried how they won't have massive latency between 100 plus clients in one node like you said.
My man, you had me at graph theory
Really interesting and excellent content, more of this please..
Great insight, I wish we had more SME's talking about the tech development.
The best quote I've taken away from a video for a long time " just like META, only not stupid"!
I have basically zero knowledge on network engineering or networks at all, but great video! It was interesting to learn a little more about the server meshing SC is always talking about!
I have no way near the experience yourself has, I have been working in networking for 10 years now, and it blows my mind what they are discussing, I cant comprehend how it will scale to the size they will require. It seems like magic to me at the moment what they are claiming is achievable, and I agree, the further implications in gaming and beyond seem interesting, tech like this in the industry I work in would be revolutionary.
great video!
"Think of this like the metaverse, except not stupid."
Gold 🤣