Oh man .. that section on "doing work without understanding" hit me so hard.... I worked a corporate job as my first job out of college and was asking questions to get understanding and all the senior level developers were just focused on the speed at which code is written and nothing else. I didn't last because I wanted to know what I was doing, not just doing...
Well sadly, something that is often not scheduled by managers is the negative impact a junior has on the productivity. More people, more work done right ? Nope. If a senior programmer has all the same amount of work expected from them, but has the extra work of training the junior, they probably wont go for a 3 hour class on how it works, but just fast forward to "just do it like that". In the end the junior programmer would get better at it and will have a positive impact on the work done, but on the short term it's a negative impact that is generally completely ignored and uncounted for by managers.
Please talk about "it's ready when it's done" and "a delayed game is eventually good", specifics on how to request more time to management and how to get them a little bit out of your back and let you work more on features/polish/qa. Thank you!
based on previous stories, i straight up think the answer is “you can’t.” Tim has mentioned a few experiences asking for more time on projects and from publishers and the response always seemed pretty negative.
As a simulation engineer I would add a caveat to “can’t optimize early” in that procedural worlds, in particular, have to be optimized at a chunk level before you scale up the world to be massive otherwise you won’t be able to test anything or even run the game. Great content and thanks for sharing your knowledge and experience for such a low price (free!)
You often hear game programmers say that it's not important to optimize early on and to wait for issues to occur later, which is weird because using optimal coding practices from the beginning doesn't take additional time, it's just a matter of choosing to do it one way vs another. There are tons of ways to optimize your game during development which will save you many headaches later on. At about 9:00 Tim talks about rewriting a function that grabs the nearest creature to the player to make it faster. Why waste time going back to rewrite functions that could have been written using optimal coding practices from the start? Sure this one function is not going to make or break a game, but if you do this enough you will get a mess.
Great talk again, Tim. A lot of my experience with optimization is similar to your example the "array of creatures." Many low level engineers focus on micro-optimization and making frequently used functions faster, but often do not step back and understand what they are actually being used for. The vast majority of optimization I've done is improving algorithms and not low level tweaking. It requires far more understanding of what exactly the game play code is doing and seeing where improvements can be made. BTW, static lighting is cheaper is actually starting to become old now. Unreal 5's Lumen is just mind blowingly fast with dynamics for example. I'm excited for the paradigm shift in graphics happening with new engines and games coming in the next few years.
You're a gosh dang wizard! this video should be watched by anyone who ever interacts with anyone near development 👀 I recall Modern Vintage Gamer did a video discussing compiler optimisation flags in his video "Did Nintendo really forget to Optimize Super Mario 64" which goes into why the US version of the game was compiled in debug mode - the common consensus at the time was that nintendo "forgot" to do it, but _of course_ it's a little bit more complicated than that
You're so important for starting devs who want to nutrish from the field experience they don't have. So grateful to have you everyday Tim, thanks a lot.
I am one of those people who have stored an entire array of nearby creatures just to find the closest, and was hoping you would tell me what was a better alternative, so thank you for that. Here's a question for you Tim... Being that junior programmer you speak of, how do you pick up the tricks towards optimization? Especially being self taught, as you say, newer programmers don't use these techniques so they're not taught in tutorials. How can I pick them up without having an experienced mentor like yourself?
This is exactly why Tim's channel is so valuable! As for tricks, many programmers develop their own tricks by digging deep: Learning how things really work under the hood. Figuring out what the problems really are under the hood. Then the trick/solution often presents itself. (googling can also help once you have enough knowledge to write the right query) As Tim said, it can take multiple tries even for the best programmers. If you don't have a senior/lead to give you the answers, focus on building strong fundamentals.
speaking of... spend all of your free time trying to answer questions on forums, you'll end up reading a lot of good answer and tricks and become confident in your own knowledge
Aw man! I AM the programner who gets all the creatures, puts them in an array, sort it, take the zero'th instance and throw it away 😅 More tips like that would be welcome!
I was just reading about an upcoming proposal in my own field of web dev that uses the dirty/clean concept. It's funny to me how different fields discover the same concepts over and over ocasionally.
When it comes to needing dynamic lights having only 1 light cast shadows while the rest don't while using AO is a great way to squeeze performance, atleast in Unreal
Or you can take the shortcut and simply switch to ray tracing to have dynamic lights for free. Future hardware will always support this. And in principle it has always been the case that old technologies are no longer used at some point. In the medium term, the problem will therefore solve itself with a greater distribution of hardware capable of fast ray tracing. At the moment, however, the market for hardware that requires classic renderers is too large not to support them, so your trick makes sense.
As Donald Knuth wrote, "The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming."
I have been putting this videos on when I get stuck or feel like I'm not making progress in my game dev/code and then something you say will randomly connect the dots and I'll figure out where I can go back and make progress. It's been a great experience, I feel like I'm giving myself time to rest and letting my brain background process while absorbing the information you're conveying in these videos. Next level workflow.
Great video! I would like more of these game programming videos ( I would also like to know more about the peculiar architecture of the ps3 or any other consoles you have developed games for in general)
I feel a common sin even among skilled people who work on performance optimization is trying to optimize code in ways or places which are not the limiting bottleneck in the first place. If your you replace a quadratic algorithm with a linear one and even vectorize it but the algorithm is called once per program on a small array in a gpu or memory bound app, it's not going to make a whole world of difference no matter how cool the algorithm is. 😅 I really love modern profiling tools and methodology of profiling (Flame graphs, VTune, RGP, etc). I can't imagine how much harder it was profiling and reasoning about code performance 30 or so years ago without these tools. (Granted, 30 years ago hardware was simpler so telling how quickly the code would run by just looking at the assembly was a bit easier? And also I was baby so my perception might be skewed.) Apologies for randomly commenting in these older videos, just really enjoying the content LOL.
Memory optimization is so important and certain huge studios seem to forget. Like my friends play that free cod game thats cross platform but for 200gb of my hard drive, screw free, you have to pay me.
Optimisation is extremely important for high-performance stuff. Games included. For casual games one would say it's less important. What would it matter if you increase the 'min requirements' in case the graphics are 'pixel-art' and the CPU has nothing to calculate anyway. The requirements were already super low anyway. The problem with optimisation by hand via assembly is the fact that different processors or architectures, or operating systems ... or anything really ... tend to have different bottlenecks. And a year after you have something released, a bottleneck can appear somewhere you had not expected it to at all, and due to Murphy's law right at the place you had exploited. For example, unraveling an additional loop with modern processors that are using predictive branching can be a problem. I remember i had been tinkering with an image processing loop in this way. The first thing to do was of course to compile and to 'steal' the initial assembly code from the compiler :-). It uses the most modern and the 'fastest' practices for ordinary stuff. Then i went to do three 'rarely triggering' branches instead of doing longer calculations and vice versa, i tried to reduce all the 'branches' to a single one by increasing the overall cycle count per loop. And then tested it on several processors. Those with 'better' branch prediction preferred more 'branches', and those who had poor branch prediction liked more cycles. Those with extensions such as SSE/AVX were not necessarily faster than their counterparts if the calculation was organised correctly (although generally SSE/AVX did boost it by a lot, but branching actually did more in my case regardless). The worst part i had figured out - in some cases you can optimise speed for a select CPU even raising it by up to 20-50%. But then you go test it on another architecture, and you get a slowdown by the same 20-50%. All because you had used some unique quirks of the first CPU you were optimising for. Technically, you could optimise for every processor and architecture on the market, but that would be super time-consuming and moreover, after the release, a new CPU will come out and that trick that saved you 20-50% of CPU time may be rendered uneffective in the new architecture, or may be will even slow it down. And the developers of said CPU/architecture would not have even planned for 'your trickery' because your solution is so fringe, it's specific for your circumstances only and is never used by anyone else. So I'm in the camp of using the predefined compiler optimisations and no assembly language checks, because it's what modern CPUs are designed for. Architectural changes to the high level code (such as the example of looking for the closest creature/NPC) are bringing much more speed/memory consumption optimisation at a lower effort. Dabble in assembly only when a bottleneck is super tight and it's really not enough. Most likely not happening in games any more at a CPU level. But the irony is - it has now sort of been reincarnated in the HLSL and GPU optimisations. There one has to optimise. I suspect less so than the last time i took a look at it (like 5-7 years ago), but i'm fairly certain AMD and NVidia are still quite different. Not to mention mobile games and the new Intel GPUs. And those new raytracing and 'directstorage' technologies that should also be quite 'optimisable'.
The youtuber Creel has done some interesting videos about that topic. In one he compared an old Athlon processor with a modern processor and showed examples within the same x86 architecture family. I can highly recommend his videos. Choosing the right algorithm at a high level has always been the best way and then towards the end, when the code is largely finished, you can use a profiler to see where it is worth optimizing by hand.
Hey Tim, I was wondering if speedrunning has any effect on the games you have made. Specifically the scope of bug fixes post launch, and internal developer views on speedrunning.
It took me a long time to realize the perils of early optimization. As I've grown I've found, in development, readability is king since you're 100% going to forget how things work. To the point that it's often worth making something less optimal if it's a lot easier to read. Once the software is finished, it's much easier to just attack your bottlenecks.
The problem why optimization isn't that big of a deal is that because of the fast processors and lots of RAM and storage space. Back in the day, it was limited, and it forced you to optimize or else it wouldn't run at an acceptable speed, not leave enough working memory for what needs to be loaded, or just not fit on the storage medium.
Greatly enjoyed this talk, thanks Tim. I've long worried that the move further and further away from assembler and reliance on more top level tools has fostered wasteful habits. There are fewer and fewer programmers out there with your pedigree, instead those with sometimes lax techniques, encouraged by management for the sake of time.
So many games theee days don't appear to optimise and jt drives me crazy, the fact you said they just start turning off flags (which I assume is removing potential optimisation improvements) makes me understand why, I guess they think it's as optimised as best it can be because now it compiles with "some" flags.
4:52 And that trend will continue (to get worse) the more people rely on pre-fab engines, etc. I saw in when I was a web-developer; I experienced it first-hand. I called the database a giant-hash-in-the-sky, until the day my performance tanked and had to hire an outside consultant.
If a game is still in "early access" after 5+ years and up to revision 15 or more the devs really need to stop adding to it and begin looking at optimisation. This does explain why some early access games I have and enjoy run poorly but have graphics that should be able to be run on a potato.
Hello, Tim. Can you, pleace explain your vision of first person fallout? You said you thought about it back in 90s. How did you imagine it? Fallout 3 and 4 wasteland is too "dense" - a village and a raiders base across the road, go 50 meters more and you find power armor in the bush, another 50 meters and a vault. On the other hand if you put a lot of empty space it will be empty like Daggerfall. So how did you saw wasteland, traveling and transition from town to wastelend? Same "green" exit fields or other way? I'd like to hear how you imagined it in 90s and how whould you make it today with modern hardware. Thank you.
Nowadays, arrays are even faster than linked lists, thanks to large cache sizes. Of course, this assumes that the entire array fits into the cache. In his example, however, his optimization is still better because it requires less calculation. It also takes time if you have to load the array into the cache first. However, optimizing for cache sizes is something you should definitely think about these days. The same goes for branchless programming.
Tim, I would really like your to hear your opinion on the modern demoscene. People there have gone as far as implement Starfox and Wolfenstein on a stock Genesis at pretty good framerates. There's also some insane stuff on the 2600 and 80s PC platforms. Granted, stuff like that really fascinates a newer programmer like me but I suspect you guys are already familiar with much of the techniques used.
Awesome! Would be great to hear how you concretely applied some techniques in Unity for Pillars of Eternity. Or for example, how do you handle data in Unity? Do you use scriptable objects, asset bundles, or some other method? Did you create many tools for the designers; which ones? Best!
A form of optimization that seems to be very undervalued these days is the size of saved games, which leads to loading time. I know that for all that I love about Pathfinder: Wrath of the Righteous, Owlcat never did very obvious ways to avoid saved game bloat. The saves track the exact state of big places (mostly remaining loot) even when we can't return to them anymore. A cleaner mod helps manage that and it can save minutes on loading games/places near the end of the campaign! Another example, if you sell vendor trash, the game will track it. You have NO reason to ever buy it back from them. That bloats the save even more.
Such a fun topic!! Thanks again for a great video! What are the tools that you have found most useful for code optimization in games you worked on? You mentioned profilers and using compiler output for identifying the right pieces of code to optimize. What other tools have you found crucial for such efforts? Thanks again!
I often push junior developers to do the opposite - avoid memoization, and compute things when they need it to avoid pitfalls of state and increase refactoring potential of the code. It depends on kind of software one writes, but i would gladly use more functional code with defined inputs and outputs, versus stateful and with a lot of implicit context through shared objects and such. Obviously there are performance critical codepaths, and all best are off there to memoize and cache computations, but in a lot of cases in my field of web apps, it's very avoidable.
Optimization depends on a lot on the systems that run it. For a Unity game for example, doing a few operations on the stack is sometimes preferable to storing something on the heap. And server side calculations with even more threads working on hundreds of requests in different states of async-await, there will be different methods to optimize. Just profile, don't guess.
These drugs in Fallout have certainly enriched the game world, but I have to admit that I have never used them in the game. Precisely because I didn't want my player character to become addicted to drugs. And that's what I do with every game. One exception was 7 Days to Die. There you could increase your stamina with beer and thus run longer and faster or carry more. It was worth it and the disadvantages of beer were bearable. But in some version they changed that so that it was no longer worth it.
sometimes just getting something to work can be hard enough. often you build something then figure out how to optimize it later. You ever try any of the Zachtronics games, like Shenzen IO or Exapunks? They are a bit like educational programming games / tutorials that help you learn about micro-controllers and assembly. There was a big thing with Arduino and raspberry pie a while ago.
No one can tell you how long it'll take to optomize code. All tasks are unique. Some you might of seen something similar before, but never in the same scenario. Appreciating O notation can be a good place to start though for juniors.
While agree with you that optimisations shouldn't be done early, what should be done from the start is writing code with enough discipline to aid in the optimization later
Is there a chance you would talk more in depth about technical side of stuff? I'm not familiar with low-level languages(yet?), I mostly write JS/TS but still I think it would be interesting and enlightening hear you talking about code architecture or handling memory etc. I understand you're mainly C/C++ programmer. Have you had experience with more modern languages which aim to improve upon those - like zig, Rust, Odin?
Hmm I wonder if a double deadline set up would help at all, setting a deadline for everything other than optimization to get done say a quarter or half a year before a release date & then spending the remaining time between that deadline & release just optimizing.
I tend to do mostly high level optimisation (algorithms, caches...) but I wonder how you tackle low level optimization in a Unity game where all you have to play with is C#. Did you have to go into low level stuff like that in your Unity games? Also, what about shader optimizations (avoiding branching and the like)?
6:15 You said the forbidden words!! The whole "we don't need to optimize because we have super computers" thing always annoys me. Yes, I get it: you want the bare minimum to run the game. But if we apply that to every app and every program, think about how much unnecessary energy is being used across the entire planet just because you didn't feel like optimizing. (And go back and click buttons on a windows XP machine vs a win11 machine and tell me which one is snappier even though modern computers are so much "faster" today.)
Can you also get into why we shouldn't optimize too early? (convincingly like the way you make the case(s) for 3rd-party vs bespoke game engines) Also the difference between writing optimal code (don't recalculate static values) vs optimization. It also feels like you're slowly developing a "Cain Explain" series for topics that are difficult to explain. lol Ex: When a producer asks "why can't you just write optimized code in the first place?" The programmer can calmly reply, "that's a great question! Tim explained it expertly in this video."
The usual explanation is that optimizing code takes time out of adding new features (Tim explained this at the beginning). It is often highly specific and also can be hard to follow and debug, leading to lost time further down the line. An example might be optimizing a ray tracing application - the basic unoptimized system will loop through all of your objects and check each for intersection with a ray. An optimized version would first of all build a (e.g.) bounding volume tree, and then traverse that tree to find the nearest intersection for the ray. Much more complex and difficult to replace or add features too, and took much longer to write. However, also note that when you come to write your next raytracer you might actually implement the BVH early because you absolutely know you will need it, in this case it is not 'too early' as it is part of your design. Low level optimization (as opposed to algorithmic) is even worse than the above, the low-hanging fruit of compiler optimizations are pretty much tapped out, mainly vectorization is still hard for compilers and writing assembly is pointless unless there are instructions that your particular language can't target. Fiddling with code to squeeze out every cycle *can* still make sense, but it must be profiled and can be a significant drain on resources both initially and during maintenance due to likely unforseen interactions.
Usually dropping features was done to hit scheduling milestones. For optimization, we rarely dropped a feature altogether, since that might require rebalancing. We did have design fallbacks though, where we replaced a feature with a similar one that was simpler and known to work…if less interesting.
I could not play the Lonesome Road DLC on my PS3. Every time I went into the Divide the frame rate dropped to less than 1 frame a second. I was soooo angry that I got rid of my collectors version of the game (which I regretted later). I still don't think I have played it since even though I multiple versions of NV on different systems. Did you play on PS3 and do you have any thoughts what could have made that happen from the PS3 architecture structure?
i feel you man I was so excited for its release and played it the moment it came out on ps3 and it would just freeze up every 5 to 10 minutes, I was so mad back then
The moment you touched any of the dlc-- the framerate plummeted and a crash would be introduced that happened every 30-60 minutes. I got around the latter by just saving often.
@redwes1, It was only the Lonesome Road DLC for me. I remember playing the others without issue aside from the normal "if you play too long in a Bethesda game it eventually crashes" thing. 😆
Your comment on static vs dynamic lighting made me recollect a recent talk on methodology given by a lead Path of Exile programmer. You might find it quite interesting (I did!). The CZcams video is titled: "The rendering techniques of Path of Exile (Alexander Sannikov)".
Hello Tim, i really enjoy your videos, they are surely inspiring and interesting at the same time! Could you make a video about your thoughts of fallout organization from fallout 1 to fallout 4, which ones you like or dislike for some reason. That would be really cool to hear your opinion on that
I do not get it! Why are you still clinging to crunch? I mean I understand that you are a freelancer and therefore you may work more than the permanent employee. If you don't know how long something will take, say so and communicate it too, e.g. "Probable date will be XY, but note that this is not certain as this implementation is new." In addition, refactoring AFTER the project is an option. I know that means you have to mess with your boss, but that's your job! I remember a situation where another department needed more time due to the cost of the development timeframe and the argument was "it takes longer, so what?". Our department head got into an argument with him, and the supervisor was upset that the two departments didn't get along and tried to find a solution. The higher your position, the more you have to deal with politics and resources. Screw crunch and do your job and if it fails then it wasn't the right project with the right people!
11:48 Are we still talking about this?? In this case the answer is: Anticipate delays. If it’s true that no programmer could ever estimate the amount of time it takes to optimize then why does MANAGEMENT pretend like we can? Why is Management and the corporate system anticipating a specific end date if they KNOW they can’t possibly estimate the amount of time certain portions of work will take? Clearly one side of this equation has an expectation that cannot be met, yet programmers are still expected to go weeks without seeing their families in service of an arbitrary deadline. Tim. No one is personally blaming you for crunch, but you have to understand 1 party says “meet this deadline, or figure out where your next meal will come from on your own” and the other party has to work as fast as possible to meet that deadline. It’s not a system worth defending anymore. There is no real ethical way to do game development as we have. Just give it up.
Setting a deadline is important because otherwise you can program forever. However, the money you have available for development is not infinite. Deadlines are therefore a guide for developers so that they can prepare for when they should be finished with something. It is inevitable that there will be delays, but a good manager would plan for this from the start and then approve more time. And of course he has to plan the latter financially in advance.
7:25 "optimisation isnt as important" ...ah, they must work on games I picked up, then droped because they were buggy messes at launch, and for the week after, and then I never picked up again! Seriously, Im old school... Take the time to release a functioning game, Devs, I dont bloody care! If it still has some minor hickups after launch, fine... bug fix patches are all good. But if an ENTIRE SYSTEM is borked at launch; or in the case of Dragon Age Inquisition, the game would KERNAL CRASH ANYONE WITH AN NVIDIA GPU... I think they need to do better. I think you either need to do better by getting the higher-ups to listen that it needs longer to cook...OR...you need to quit and find another job at a company where you CAN tell them that. Personally, I think Obsidian is good when its THEM...just them. When its not...well...FO:NV, KOTOR2, etc... but when it IS just them...PoE/2, Tyranny...I mean, you guys do good work when you set reasonable timetables to do the work, test it, and get it right. And Tim, I love this "behind the curtain" scoop you give us.
Its not devs that push for deadlines, its publishers and first party studios. And time is money so they shave every minute off of production they can and studios have to work with it.
That's not optimization you're talking about, that's just QA & debugging. QA needs some love. A lot. But graphics cards... There are so many that most studios can't hope to test more than 1 or 2% of these.
I’ve only done a tiny bit of game coding and immediately fucked up everything by writing horribly inefficient code. I can’t imagine how hard it must be to optimize a real commercial game.
I don't think people fully realize what a gem this channel is if you're a young, budding programmer/game dev.
So true! Just started sixth page of notes lol. This series helps a lot with the stuff with the foundational stuff you need
What about sub-middle-aged budding programmers /possible game devs?
@@Rockyzach88 I don't see age being a restriction!
Oh man .. that section on "doing work without understanding" hit me so hard....
I worked a corporate job as my first job out of college and was asking questions to get understanding and all the senior level developers were just focused on the speed at which code is written and nothing else.
I didn't last because I wanted to know what I was doing, not just doing...
> senior level developers
'Senior'. Boot-licking good 'ole boys. Yeah I'm mad about it.
Those Senior devs are nothing more than stuck up cunts if they didn't even bother explaining anything to a junior dev
Yeah this rings very true, unfortunately.
Well sadly, something that is often not scheduled by managers is the negative impact a junior has on the productivity.
More people, more work done right ? Nope. If a senior programmer has all the same amount of work expected from them, but has the extra work of training the junior, they probably wont go for a 3 hour class on how it works, but just fast forward to "just do it like that".
In the end the junior programmer would get better at it and will have a positive impact on the work done, but on the short term it's a negative impact that is generally completely ignored and uncounted for by managers.
Please talk about "it's ready when it's done" and "a delayed game is eventually good", specifics on how to request more time to management and how to get them a little bit out of your back and let you work more on features/polish/qa.
Thank you!
How to revise scope and reprioritize to compensate for the differences between estimation vs reality.
based on previous stories, i straight up think the answer is “you can’t.” Tim has mentioned a few experiences asking for more time on projects and from publishers and the response always seemed pretty negative.
As a simulation engineer I would add a caveat to “can’t optimize early” in that procedural worlds, in particular, have to be optimized at a chunk level before you scale up the world to be massive otherwise you won’t be able to test anything or even run the game.
Great content and thanks for sharing your knowledge and experience for such a low price (free!)
_“Premature optimization is the root of all evil”_
- Donald Knuth
every computer science student knows this quote
i didnt :(
You often hear game programmers say that it's not important to optimize early on and to wait for issues to occur later, which is weird because using optimal coding practices from the beginning doesn't take additional time, it's just a matter of choosing to do it one way vs another. There are tons of ways to optimize your game during development which will save you many headaches later on. At about 9:00 Tim talks about rewriting a function that grabs the nearest creature to the player to make it faster. Why waste time going back to rewrite functions that could have been written using optimal coding practices from the start? Sure this one function is not going to make or break a game, but if you do this enough you will get a mess.
Great talk again, Tim. A lot of my experience with optimization is similar to your example the "array of creatures." Many low level engineers focus on micro-optimization and making frequently used functions faster, but often do not step back and understand what they are actually being used for. The vast majority of optimization I've done is improving algorithms and not low level tweaking. It requires far more understanding of what exactly the game play code is doing and seeing where improvements can be made.
BTW, static lighting is cheaper is actually starting to become old now. Unreal 5's Lumen is just mind blowingly fast with dynamics for example. I'm excited for the paradigm shift in graphics happening with new engines and games coming in the next few years.
It's not the engines, it's ray tracing. Dynamic lights come free with ray traycing.
You're a gosh dang wizard! this video should be watched by anyone who ever interacts with anyone near development 👀
I recall Modern Vintage Gamer did a video discussing compiler optimisation flags in his video "Did Nintendo really forget to Optimize Super Mario 64" which goes into why the US version of the game was compiled in debug mode - the common consensus at the time was that nintendo "forgot" to do it, but _of course_ it's a little bit more complicated than that
Those Super Mario 64 videos are fascinating.
You're so important for starting devs who want to nutrish from the field experience they don't have. So grateful to have you everyday Tim, thanks a lot.
your show is my favorite thing in youtube in the last month
I am one of those people who have stored an entire array of nearby creatures just to find the closest, and was hoping you would tell me what was a better alternative, so thank you for that.
Here's a question for you Tim... Being that junior programmer you speak of, how do you pick up the tricks towards optimization? Especially being self taught, as you say, newer programmers don't use these techniques so they're not taught in tutorials. How can I pick them up without having an experienced mentor like yourself?
This is exactly why Tim's channel is so valuable!
As for tricks, many programmers develop their own tricks by digging deep:
Learning how things really work under the hood.
Figuring out what the problems really are under the hood.
Then the trick/solution often presents itself. (googling can also help once you have enough knowledge to write the right query)
As Tim said, it can take multiple tries even for the best programmers. If you don't have a senior/lead to give you the answers, focus on building strong fundamentals.
asking this on stack overflow would already have you an answer and you'll be one step closer to being a rockstar
speaking of... spend all of your free time trying to answer questions on forums, you'll end up reading a lot of good answer and tricks and become confident in your own knowledge
Aw man! I AM the programner who gets all the creatures, puts them in an array, sort it, take the zero'th instance and throw it away 😅
More tips like that would be welcome!
I was just reading about an upcoming proposal in my own field of web dev that uses the dirty/clean concept.
It's funny to me how different fields discover the same concepts over and over ocasionally.
Marking stuff as dirty is the bread and butter of game devs haha
Thanks as always for the videos, loving the programming talks.
When it comes to needing dynamic lights having only 1 light cast shadows while the rest don't while using AO is a great way to squeeze performance, atleast in Unreal
Or you can take the shortcut and simply switch to ray tracing to have dynamic lights for free. Future hardware will always support this. And in principle it has always been the case that old technologies are no longer used at some point. In the medium term, the problem will therefore solve itself with a greater distribution of hardware capable of fast ray tracing. At the moment, however, the market for hardware that requires classic renderers is too large not to support them, so your trick makes sense.
I liked this one because you gave examples of actual algorithms/code you use in creating games or at least a summary of what it was.
"I caught a fish THIS big"- Timothy Cain
You are reminding me of my Gentoo days where people would put -funroll-loops in their CFLAGS to make it "fastah".
As Donald Knuth wrote, "The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming."
I have been putting this videos on when I get stuck or feel like I'm not making progress in my game dev/code and then something you say will randomly connect the dots and I'll figure out where I can go back and make progress.
It's been a great experience, I feel like I'm giving myself time to rest and letting my brain background process while absorbing the information you're conveying in these videos.
Next level workflow.
Great video! I would like more of these game programming videos ( I would also like to know more about the peculiar architecture of the ps3 or any other consoles you have developed games for in general)
I feel a common sin even among skilled people who work on performance optimization is trying to optimize code in ways or places which are not the limiting bottleneck in the first place. If your you replace a quadratic algorithm with a linear one and even vectorize it but the algorithm is called once per program on a small array in a gpu or memory bound app, it's not going to make a whole world of difference no matter how cool the algorithm is. 😅
I really love modern profiling tools and methodology of profiling (Flame graphs, VTune, RGP, etc). I can't imagine how much harder it was profiling and reasoning about code performance 30 or so years ago without these tools. (Granted, 30 years ago hardware was simpler so telling how quickly the code would run by just looking at the assembly was a bit easier? And also I was baby so my perception might be skewed.)
Apologies for randomly commenting in these older videos, just really enjoying the content LOL.
Memory optimization is so important and certain huge studios seem to forget. Like my friends play that free cod game thats cross platform but for 200gb of my hard drive, screw free, you have to pay me.
Optimisation is extremely important for high-performance stuff. Games included. For casual games one would say it's less important. What would it matter if you increase the 'min requirements' in case the graphics are 'pixel-art' and the CPU has nothing to calculate anyway. The requirements were already super low anyway.
The problem with optimisation by hand via assembly is the fact that different processors or architectures, or operating systems ... or anything really ... tend to have different bottlenecks. And a year after you have something released, a bottleneck can appear somewhere you had not expected it to at all, and due to Murphy's law right at the place you had exploited.
For example, unraveling an additional loop with modern processors that are using predictive branching can be a problem. I remember i had been tinkering with an image processing loop in this way. The first thing to do was of course to compile and to 'steal' the initial assembly code from the compiler :-). It uses the most modern and the 'fastest' practices for ordinary stuff. Then i went to do three 'rarely triggering' branches instead of doing longer calculations and vice versa, i tried to reduce all the 'branches' to a single one by increasing the overall cycle count per loop. And then tested it on several processors. Those with 'better' branch prediction preferred more 'branches', and those who had poor branch prediction liked more cycles. Those with extensions such as SSE/AVX were not necessarily faster than their counterparts if the calculation was organised correctly (although generally SSE/AVX did boost it by a lot, but branching actually did more in my case regardless).
The worst part i had figured out - in some cases you can optimise speed for a select CPU even raising it by up to 20-50%. But then you go test it on another architecture, and you get a slowdown by the same 20-50%. All because you had used some unique quirks of the first CPU you were optimising for. Technically, you could optimise for every processor and architecture on the market, but that would be super time-consuming and moreover, after the release, a new CPU will come out and that trick that saved you 20-50% of CPU time may be rendered uneffective in the new architecture, or may be will even slow it down. And the developers of said CPU/architecture would not have even planned for 'your trickery' because your solution is so fringe, it's specific for your circumstances only and is never used by anyone else.
So I'm in the camp of using the predefined compiler optimisations and no assembly language checks, because it's what modern CPUs are designed for. Architectural changes to the high level code (such as the example of looking for the closest creature/NPC) are bringing much more speed/memory consumption optimisation at a lower effort. Dabble in assembly only when a bottleneck is super tight and it's really not enough. Most likely not happening in games any more at a CPU level. But the irony is - it has now sort of been reincarnated in the HLSL and GPU optimisations. There one has to optimise. I suspect less so than the last time i took a look at it (like 5-7 years ago), but i'm fairly certain AMD and NVidia are still quite different. Not to mention mobile games and the new Intel GPUs. And those new raytracing and 'directstorage' technologies that should also be quite 'optimisable'.
The youtuber Creel has done some interesting videos about that topic. In one he compared an old Athlon processor with a modern processor and showed examples within the same x86 architecture family. I can highly recommend his videos. Choosing the right algorithm at a high level has always been the best way and then towards the end, when the code is largely finished, you can use a profiler to see where it is worth optimizing by hand.
Loves these kinds of videos, keep em coming!
This is one of my favorite subjects i learned in my Datastructures and Algorithms class, Polynomial, Non-Polynomial, and Non-Polynomial Hard.
Hey Tim, I was wondering if speedrunning has any effect on the games you have made. Specifically the scope of bug fixes post launch, and internal developer views on speedrunning.
Great question I’d love a video on non dev supported modes of play like that
It took me a long time to realize the perils of early optimization. As I've grown I've found, in development, readability is king since you're 100% going to forget how things work. To the point that it's often worth making something less optimal if it's a lot easier to read. Once the software is finished, it's much easier to just attack your bottlenecks.
The problem why optimization isn't that big of a deal is that because of the fast processors and lots of RAM and storage space. Back in the day, it was limited, and it forced you to optimize or else it wouldn't run at an acceptable speed, not leave enough working memory for what needs to be loaded, or just not fit on the storage medium.
The way you describe how some programmers are using inefficient code because its easy both makes me sad and matches my 20 year gaming experience
Without modding, the Boston downtown area in Fallout 4 runs poorly even on supercomputers and it's a darn shame.
It runs at a stable 30fps on my ps5 with the previous gen version
@@packrunnernesyes, that is poorly. We want 100fps+ when playing on good (PC) hardware.
@@Jwlar it's not poorly, it only runs at 30 because it was locked at 30
@@packrunnernes It could be runing at 34 for all you know
@@Jwlar all I'm trying to say, if it can run fine on a ps5 I'm pretty sure it can run well on "supercomputers"
off topic but that tee looks awesome!
Greatly enjoyed this talk, thanks Tim. I've long worried that the move further and further away from assembler and reliance on more top level tools has fostered wasteful habits. There are fewer and fewer programmers out there with your pedigree, instead those with sometimes lax techniques, encouraged by management for the sake of time.
I love that Obsidian shirt so much!
wow, this video finally taught me wtf pragma means/does lol.
So many games theee days don't appear to optimise and jt drives me crazy, the fact you said they just start turning off flags (which I assume is removing potential optimisation improvements) makes me understand why, I guess they think it's as optimised as best it can be because now it compiles with "some" flags.
Things I'm going to put on my list of rabbit holes to fall down: Application Profiling and Optimization theory and practice.
Holy god!
I need that shirt!
4:52 And that trend will continue (to get worse) the more people rely on pre-fab engines, etc. I saw in when I was a web-developer; I experienced it first-hand. I called the database a giant-hash-in-the-sky, until the day my performance tanked and had to hire an outside consultant.
If a game is still in "early access" after 5+ years and up to revision 15 or more the devs really need to stop adding to it and begin looking at optimisation. This does explain why some early access games I have and enjoy run poorly but have graphics that should be able to be run on a potato.
Hello, Tim. Can you, pleace explain your vision of first person fallout? You said you thought about it back in 90s. How did you imagine it? Fallout 3 and 4 wasteland is too "dense" - a village and a raiders base across the road, go 50 meters more and you find power armor in the bush, another 50 meters and a vault. On the other hand if you put a lot of empty space it will be empty like Daggerfall. So how did you saw wasteland, traveling and transition from town to wastelend? Same "green" exit fields or other way? I'd like to hear how you imagined it in 90s and how whould you make it today with modern hardware. Thank you.
Recalculating a value in the CPU can be quicker if the alternative is a slow memory lookup. Its really something that would needs to be tested.
Nowadays, arrays are even faster than linked lists, thanks to large cache sizes. Of course, this assumes that the entire array fits into the cache. In his example, however, his optimization is still better because it requires less calculation. It also takes time if you have to load the array into the cache first. However, optimizing for cache sizes is something you should definitely think about these days. The same goes for branchless programming.
Tim, I would really like your to hear your opinion on the modern demoscene. People there have gone as far as implement Starfox and Wolfenstein on a stock Genesis at pretty good framerates. There's also some insane stuff on the 2600 and 80s PC platforms.
Granted, stuff like that really fascinates a newer programmer like me but I suspect you guys are already familiar with much of the techniques used.
Awesome!
Would be great to hear how you concretely applied some techniques in Unity for Pillars of Eternity. Or for example, how do you handle data in Unity? Do you use scriptable objects, asset bundles, or some other method? Did you create many tools for the designers; which ones?
Best!
A form of optimization that seems to be very undervalued these days is the size of saved games, which leads to loading time. I know that for all that I love about Pathfinder: Wrath of the Righteous, Owlcat never did very obvious ways to avoid saved game bloat. The saves track the exact state of big places (mostly remaining loot) even when we can't return to them anymore. A cleaner mod helps manage that and it can save minutes on loading games/places near the end of the campaign! Another example, if you sell vendor trash, the game will track it. You have NO reason to ever buy it back from them. That bloats the save even more.
That's a feature you see...
It teaches us the importance of not littering.
Such a fun topic!! Thanks again for a great video!
What are the tools that you have found most useful for code optimization in games you worked on? You mentioned profilers and using compiler output for identifying the right pieces of code to optimize. What other tools have you found crucial for such efforts?
Thanks again!
I often push junior developers to do the opposite - avoid memoization, and compute things when they need it to avoid pitfalls of state and increase refactoring potential of the code. It depends on kind of software one writes, but i would gladly use more functional code with defined inputs and outputs, versus stateful and with a lot of implicit context through shared objects and such. Obviously there are performance critical codepaths, and all best are off there to memoize and cache computations, but in a lot of cases in my field of web apps, it's very avoidable.
Optimization depends on a lot on the systems that run it. For a Unity game for example, doing a few operations on the stack is sometimes preferable to storing something on the heap. And server side calculations with even more threads working on hundreds of requests in different states of async-await, there will be different methods to optimize. Just profile, don't guess.
I would love to hear you talk about the South Park game some more, and the extent of your envolvement in it
Awesome video. Would love to hear your thoughts on Unity DOTS/ECS for optimization
These drugs in Fallout have certainly enriched the game world, but I have to admit that I have never used them in the game. Precisely because I didn't want my player character to become addicted to drugs. And that's what I do with every game. One exception was 7 Days to Die. There you could increase your stamina with beer and thus run longer and faster or carry more. It was worth it and the disadvantages of beer were bearable. But in some version they changed that so that it was no longer worth it.
I find amazing how a "PC" made game performance way better. I have a PC with no GPU that runs Battlefield 4 but suffers to run Dark Souls.
sometimes just getting something to work can be hard enough. often you build something then figure out how to optimize it later. You ever try any of the Zachtronics games, like Shenzen IO or Exapunks? They are a bit like educational programming games / tutorials that help you learn about micro-controllers and assembly. There was a big thing with Arduino and raspberry pie a while ago.
No one can tell you how long it'll take to optomize code. All tasks are unique. Some you might of seen something similar before, but never in the same scenario. Appreciating O notation can be a good place to start though for juniors.
8:45 What if player moved? The distance would also change (to all creatures).
While agree with you that optimisations shouldn't be done early, what should be done from the start is writing code with enough discipline to aid in the optimization later
Zeroth, oneth, twoth, that's how you count an array!
Is there a chance you would talk more in depth about technical side of stuff? I'm not familiar with low-level languages(yet?), I mostly write JS/TS but still I think it would be interesting and enlightening hear you talking about code architecture or handling memory etc.
I understand you're mainly C/C++ programmer. Have you had experience with more modern languages which aim to improve upon those - like zig, Rust, Odin?
Hmm I wonder if a double deadline set up would help at all, setting a deadline for everything other than optimization to get done say a quarter or half a year before a release date & then spending the remaining time between that deadline & release just optimizing.
Theirs a good story on CZcams about how a Game company optimized doom 3 to run and get released on the original Xbox.
I tend to do mostly high level optimisation (algorithms, caches...) but I wonder how you tackle low level optimization in a Unity game where all you have to play with is C#. Did you have to go into low level stuff like that in your Unity games?
Also, what about shader optimizations (avoiding branching and the like)?
6:15 You said the forbidden words!!
The whole "we don't need to optimize because we have super computers" thing always annoys me.
Yes, I get it: you want the bare minimum to run the game. But if we apply that to every app and every program, think about how much unnecessary energy is being used across the entire planet just because you didn't feel like optimizing.
(And go back and click buttons on a windows XP machine vs a win11 machine and tell me which one is snappier even though modern computers are so much "faster" today.)
Nowadays when I read "Optimization" in a post release patch note, i fear it goes along with a graphical downgrade in some sorts.
Tim's brain is gourmet cuisine.
optimizing on the assembler level seems a bit overkill in today's context, especially when you're working with a third party engine.
My favourite is seeing questions about optimization on stack overflow and some clown replies by quoting Knuth out of context.
opTIMization you could say
I wonder if Cell in ps3 not stopping related to why modern games don't use cores more efficiently.
Before view my guess is optimization is using fewer calculations for a similar enough end result.
Let's see if I'm right.
Have you ever played a portion of a game and thought to yourself "Oh this must have been a nightmare to optimize"
Can you also get into why we shouldn't optimize too early? (convincingly like the way you make the case(s) for 3rd-party vs bespoke game engines)
Also the difference between writing optimal code (don't recalculate static values) vs optimization.
It also feels like you're slowly developing a "Cain Explain" series for topics that are difficult to explain. lol
Ex: When a producer asks "why can't you just write optimized code in the first place?" The programmer can calmly reply, "that's a great question! Tim explained it expertly in this video."
The usual explanation is that optimizing code takes time out of adding new features (Tim explained this at the beginning). It is often highly specific and also can be hard to follow and debug, leading to lost time further down the line. An example might be optimizing a ray tracing application - the basic unoptimized system will loop through all of your objects and check each for intersection with a ray. An optimized version would first of all build a (e.g.) bounding volume tree, and then traverse that tree to find the nearest intersection for the ray. Much more complex and difficult to replace or add features too, and took much longer to write. However, also note that when you come to write your next raytracer you might actually implement the BVH early because you absolutely know you will need it, in this case it is not 'too early' as it is part of your design.
Low level optimization (as opposed to algorithmic) is even worse than the above, the low-hanging fruit of compiler optimizations are pretty much tapped out, mainly vectorization is still hard for compilers and writing assembly is pointless unless there are instructions that your particular language can't target. Fiddling with code to squeeze out every cycle *can* still make sense, but it must be profiled and can be a significant drain on resources both initially and during maintenance due to likely unforseen interactions.
Tim did you ever optimise just by dropping features?
Usually dropping features was done to hit scheduling milestones. For optimization, we rarely dropped a feature altogether, since that might require rebalancing. We did have design fallbacks though, where we replaced a feature with a similar one that was simpler and known to work…if less interesting.
I made a cos sin table for Gameboy. Because it did not have cos or sin calculation. We use unsigned char instead of int.
I like you!
I could not play the Lonesome Road DLC on my PS3. Every time I went into the Divide the frame rate dropped to less than 1 frame a second. I was soooo angry that I got rid of my collectors version of the game (which I regretted later). I still don't think I have played it since even though I multiple versions of NV on different systems. Did you play on PS3 and do you have any thoughts what could have made that happen from the PS3 architecture structure?
i feel you man
I was so excited for its release and played it the moment it came out on ps3 and it would just freeze up every 5 to 10 minutes, I was so mad back then
The moment you touched any of the dlc-- the framerate plummeted and a crash would be introduced that happened every 30-60 minutes. I got around the latter by just saving often.
@redwes1, It was only the Lonesome Road DLC for me. I remember playing the others without issue aside from the normal "if you play too long in a Bethesda game it eventually crashes" thing. 😆
Hey, would you ever teach programming to new devs?
Your comment on static vs dynamic lighting made me recollect a recent talk on methodology given by a lead Path of Exile programmer. You might find it quite interesting (I did!). The CZcams video is titled: "The rendering techniques of Path of Exile (Alexander Sannikov)".
Hello Tim, i really enjoy your videos, they are surely inspiring and interesting at the same time! Could you make a video about your thoughts of fallout organization from fallout 1 to fallout 4, which ones you like or dislike for some reason. That would be really cool to hear your opinion on that
Hi, Tim.
Are you allowed to show any of the six Fallout demos, if you still have any of those builds?
I do not get it! Why are you still clinging to crunch?
I mean I understand that you are a freelancer and therefore you may work more than the permanent employee.
If you don't know how long something will take, say so and communicate it too, e.g. "Probable date will be XY, but note that this is not certain as this implementation is new."
In addition, refactoring AFTER the project is an option. I know that means you have to mess with your boss, but that's your job!
I remember a situation where another department needed more time due to the cost of the development timeframe and the argument was "it takes longer, so what?". Our department head got into an argument with him, and the supervisor was upset that the two departments didn't get along and tried to find a solution.
The higher your position, the more you have to deal with politics and resources. Screw crunch and do your job and if it fails then it wasn't the right project with the right people!
pillars of eternity wasn't particularly well optimized, unlike temple of elemental evil
I straight up don't play Fallout 4 because it's so disgustingly optimized. A poorly optimized game is a bad game, no matter how fun the gameplay is.
11:48 Are we still talking about this??
In this case the answer is: Anticipate delays.
If it’s true that no programmer could ever estimate the amount of time it takes to optimize then why does MANAGEMENT pretend like we can? Why is Management and the corporate system anticipating a specific end date if they KNOW they can’t possibly estimate the amount of time certain portions of work will take? Clearly one side of this equation has an expectation that cannot be met, yet programmers are still expected to go weeks without seeing their families in service of an arbitrary deadline.
Tim. No one is personally blaming you for crunch, but you have to understand 1 party says “meet this deadline, or figure out where your next meal will come from on your own” and the other party has to work as fast as possible to meet that deadline. It’s not a system worth defending anymore. There is no real ethical way to do game development as we have. Just give it up.
Setting a deadline is important because otherwise you can program forever. However, the money you have available for development is not infinite. Deadlines are therefore a guide for developers so that they can prepare for when they should be finished with something. It is inevitable that there will be delays, but a good manager would plan for this from the start and then approve more time. And of course he has to plan the latter financially in advance.
@@OpenGL4ever yeah we agree!
7:25 "optimisation isnt as important"
...ah, they must work on games I picked up, then droped because they were buggy messes at launch, and for the week after, and then I never picked up again!
Seriously,
Im old school...
Take the time to release a functioning game, Devs, I dont bloody care! If it still has some minor hickups after launch, fine... bug fix patches are all good. But if an ENTIRE SYSTEM is borked at launch; or in the case of Dragon Age Inquisition, the game would KERNAL CRASH ANYONE WITH AN NVIDIA GPU... I think they need to do better. I think you either need to do better by getting the higher-ups to listen that it needs longer to cook...OR...you need to quit and find another job at a company where you CAN tell them that.
Personally, I think Obsidian is good when its THEM...just them.
When its not...well...FO:NV, KOTOR2, etc... but when it IS just them...PoE/2, Tyranny...I mean, you guys do good work when you set reasonable timetables to do the work, test it, and get it right.
And Tim, I love this "behind the curtain" scoop you give us.
Its not devs that push for deadlines, its publishers and first party studios. And time is money so they shave every minute off of production they can and studios have to work with it.
That's not optimization you're talking about, that's just QA & debugging.
QA needs some love. A lot.
But graphics cards... There are so many that most studios can't hope to test more than 1 or 2% of these.
All programmers know for the fact that premature optimization is THE root of all evil. Do not do this.
But if it lags to the point where you cant playtest then maybe just a little optimisation is required
I’ve only done a tiny bit of game coding and immediately fucked up everything by writing horribly inefficient code. I can’t imagine how hard it must be to optimize a real commercial game.
PLEASE PLEASE PLEASE Talk about Starfield vs The Outer Worlds
Both have nothing in common besides being set in space
@hisscivilization also, to double down on this, Tim doesn't work for Bethesda and has stated numerous times he doesn't review games.
Here we go again with crunch 😂 it's nobody's fault you were a sucky manager other than your own 👍🏼
Did you work under Tim?
@@ArchOfficial no
@@adan6566 So you're making a value statement about his managerial skills and asserting it as fact without having experience with it?
@@ArchOfficial yes
@@adan6566 Yeah, so people who read your comment should probably ignore it.