3D Gaussian Splatting in UNREAL ENGINE 5 is INSANE!
Vložit
- čas přidán 28. 05. 2024
- 3D Gaussian Splatting is one of the MOST PHOTOREALISTIC methods to reconstruct our world in 3D. Now we've done the tests but its no good till we bring them into Unreal Engine 5 to see if we can use it for different purposes. We are doing that using a plugin which is available on Unreal Engine Marketplace. Tested the quality, performance, crop tools and even set up some cameras and even made some tiny cinematic videos.
3D Gaussian Splatting for Real-Time Radiance Field Rendering Paper:
repo-sam.inria.fr/fungraph/3d...
Unreal Engine 5 Gaussian Splatting Plugin Link:
www.unrealengine.com/marketpl...
Shoutout to Maged for the Pyramids Drone footage
Join our discord server:
/ discord
If you wanna see us to do cool things follow us here too:
Instagram: / badxstudio
Twitter: / badxstudio
TikTok: / badxstudio
LinkedIn: / badxstudio
Bad Decisions Podcast 🎙️:
podcasters.spotify.com/pod/sh...
Our personal handles: (if you wanna stalk us)
/ farhad_sh
/ farazshababs
/ farhads__
/ farazshababi
/ farhadshababi
/ farazshababi
#unrealengine5 #3dgaussiansplatting #3dscan #gaussiansplatting #3d #3drender #nerf #unrealengine #blender3d #blender #drone #ai #photogrammetry #vfx #cgi - Zábava
That Unity joke made me fall out of my seat laughing. Super relevant right now.
Hahahahha exactly why that came :D Honestly that was improv not scripted :D
..you fell out of your chair laughing? It was THAT funny?
@@mikda360 a lil bit =)
I like Unity too, but that joke is badass!!
YES! Thank you for going live and explaining some of the questions! you guys ROCK!
No worries brother
Looking forward to your next video on G-Splats! So fun to see you experimenting with it.
Thank you Joel! As we speak we have at least 3 more Gsplat videos we are working on so stay tuned
I've been waiting for something like this to happen. What a time to be alive!
It's crazy
ehe, is this a quote from TwoMinutePapers?
Respect. I think you guys are following into the footstep of the Corridor Crew. I like the flavor that you bring with the video scripts/montage. Subscribed.
Thanks for the sub Tariq
Cool! Very nice video again! It was interesting to see how Kenjis UE plugin runs in practice. Great job!
Dude thank you
Thx guys ! Looking forward seeing where all of these new techs lead us
You are absolutely welcome and damn are excited about it too!
This just keeps getting better and better!!
It's going fasttt
Really great video, I’m definitely subscribed to continue having fun with you guys! 😂
hahahha man it's gonna be amazing
You guys are the best, literally! Big hug from Chile!! I will try GSplats this weekend for the terrain of my architecture thesis project, to see if next week I can bring all together to UE5. Hope everything works :)
Dudeeee big hugs to you! oooooh that terrain idea would be so cool! Would love to see how that turns out. Bring it on in our discord
Exactly what I'm going to use this for. Maybe not in Unreal but web-based
Dude!!! This is SO COOL!!!! I can't believe how far back the detail goes
It's mind blowing bro.
thanks for bringing interest to these new/old techniques!! :)
Glad you like them!
Yessss.!!!!Awesome!!!Thank you for testing this app. Now I buy Thanks!
More Gaussian!!!!
Pretty awesome video, thanks for sharing!
No worries 🖤
Very exciting! Have you done any side-by-side tests on if shooting 1080 vs 4K vs 50MP stills vs HRDI vs RAW give and significant quality advantages in the final g-splat?
That's a brilliant video idea we are destined to try! The next few videos are going to be slightly different tho but its coming for sure.
Hint for next video: Easiest Method to do 3D Gaussian Splats!
for certain applications, you could make crazy games like this
Guys, please, try this in VR! UE+3GS+VR it will be mind-blowing, I sure 🙌🏻
Yeah buddy, that will be the next experiment ...we have them all lined up.. Virtual Production, VR and etc.
I love the enthusiasm of you guys. You keep inspiring me to learn and try new things.
Great to hear that buddy 🙌🙌
Okay so, hear me out. As an artist, a traditionally trained painter and one who's pretty familiar with 2D digital illustration on a tablet, this has me imagining a new level of digital painting tool. Instead of working on a tablet, you'd wear a VR headset and paint splats directly into three dimensional space with a motion controller.
Imagine a 3D rendered scene in Unreal, that's hand painted into space and looks like an impressionist painting. I want that. I want to be able to do that.
Wow that sounds absolutely nuts! That would look soooo coool!
i think you can do that in dreams for ps4/ps5
I wanna use this to keep the same camera angle i intended but stabilize shaky shots.
I'm a freak of stabilization and this seems like the ultimate solution.
maybe some basic masking the 2 methods might fix the sky? or the colors will differ too much?
nice work keep it up.
Hi there. Nice vid but I struggle with a small detail. Maybe you can give me a hint? How do you link the gauss to a cinema camera actor instead of the viewport to render out?
Amazing content! Keep it up!
Appreciate it!
Looking forward to making my house and neighborhood a map in a future video game with like a minute of video from my phone. This is insanely cool.
Hello guys, first I want to say amazing work as always!
Do you know if there are other methods of cleaning your splat aside from cropping with the plugin? Is it possible to do it with another 3d software like blender?
Thanks for hearing me out!
Totally blown away!!
Right??!?
How are you able to get the GS visible in the CineCameraActor for the rendered cinematics. When I look through my camera the GS is not visible even when the GS is directly in front of the camera?
Gaussian splatting can be the future of 3d rendering
We are going full speed!!
Also gaming industry
excellent video! What is the name of the program to preview the frame sequence, when you drag the mouse, move forward or backward in time?
DJV .. you will love it :)
Thanks a lot for sharing this !
I was very impatient to find a way to get it in UE (I was searching for it, found your video)
I hope we will get it Twinmotion as well in a not too far future...
And of course to be able to edit the splats !!! That would be great !
No worries at all! Yesss if we can edit the splats it would be such a game changer!!!!!
This is getting crazy!!!
Completely out of control haha
Heads up, there is a new plugin out there now that has supports importing edited PLY files, so you don't have to deal with floaters, objects, distortions and areas that you don't want. I can't post link because of CZcams's horrible filtering, but there is literally only two plugins on Marketplace right now, so :P
whats the name
@@demonhogo Just search for them. Again, there are only two 3D gaussian splatting plugins on the store. Also, Super-Splat now can export to native 3DGS PLY, so you don't need compatibility plugins anymore, all plugins will work, including the free ones.
pretty cool vid, thanks.
i m new to this channel but it sounds you guys come from film. i'm a gamedev, so i see it from that pov.
i find Gaussian Splatting interesting, though i really think mesh reconstruction with nanite could give better results, and speed. specially on the text of the signs (depending on what you want to do). (also reflections, light and collisions).
though i'm actually interested to know if actually the performance diff between both.
if GS are 100% on niagara it might outperform nanite (given it might be simpler).
though on a game, i wouldn't want to use up all my gpu with GS, the game itself would be demanding.
It would be nice if you guys could share the gpu consumption of playing in the scene. i wonder if you could do stuff in VR like virtual stuff (like a "walk around somewhere" or "see this stuff").
i think the cool stuff on GS was the part where it captured the "floaters" i think that makes for cool clouds. but i think volumetric clouds are actually better, though maybe less realistic.
another point might be how it works with collisions lights and reflections. i think GS visualization might be much less flexible in that regard.
i really hope Epic adds GS directly to the engine, it might just be a cool thing to mix with other pieces. Maybe for certain things, like an intro animation (like FFVII e.g.), or mountains on the distance or buildings.
I wonder how well it could render a scene if you got more scans closer to parts of objects and more angles too! This has a lot of potential
Probably more detail then? I mean this is all still too early! But have u seen our newest video? We even turned movies to 3D with this
I think that the issue with the sky generating weird gaussians could be remedied if the software knew what certain parts of the images are supposed be like. For example you could mark parts of the image and tag them as "sky" or "far away background" and it would then know to generate those splats far away instead of how it is now. Someone would just have to code in that functionality first. :D
I hope something like that happens ASAP! NEED IT!!!
Hey, which software are you guys using for video playback?
I'm curious if one can do in-game rendering with this.
Like do (2 or more)-stereopair for inner rendering -> do GS estimation -> display rendering with GS.
Some optimizations come to mind. As I can get GS can allow some extrapolation on multiple frames for static scene when view camera moves.
So one can do inner rendering less frequently.
For GSs I can open in the python view they don't show up one I import with the plugin and drag into scene with a light. Is there any niagra prerequisites which need installed? It shows when I edit the BP but not in a blank scene, even scaled large and small. Any sanity checks I can perform? Geo shows up fine in the scene, just not these BPs it creates in a blank scene.
Need some details video how you did it, cabt understand how that splater came or was processed , really confused that how the plugin dose it, dose it take image sequence directly and dose it or we need other software to make the gaussian spaltter ???
i wanna know too, did you find anything about it?
This is very nice.
7:21: that Chrono Trigger "End of Time" vibe
Lololol true!
It seems this technology could also serve as the perfect LOD bias setting, instead of switching out for a lower poly model you can just switch to the Gaussian representation.
I imagine it will be very useful for open world games, like assassins creed or cyberpunk 2077.
It seems that with the combination of these technologies with things like DLSS and Gaussian rendering we can get games to both look better and run better than native, but easier said than done lol.
This is dope! Just learning about this new tech as I would like to find ways to innovate with my drone work in the 3D world 😊.
Btw if you want to use some footage from Puerto Rico for these videos I’ll gladly provide one just to see you do some magic with it 🙂
Keep up the good stuff! 🍻
Hey broski! Love it. Sure thing you can always join our discord and share it with us there
Imagine Microsoft integrating this into Flight Simulator. It's already scary realistic looking at times. With this, it would become indistinguishable from real life.
This as a tool in creating movies, shows and I can even see plenty ways to make this useful in 3D games, if it really takes down the processing power to achieve this level of realism, it's going to be a big thing in all media going forward.
One issue I see though, the lighting and shadows from the time of day you record in are baked into it. To make this useful in games, you'll need to figure ways to convert this to a neutral lighting, so you can apply lighting to it.
100% agreed. This is still at a very early stage and we already have Unity and UE Plugin. Probably there will be updates to lighting and shadows in a very near future
Yall innovating at record speeds
Hehehe jus wait till you see the next videos!!
Nice!!!
Supercooool!!! Would it be possible to grab an hdri with rendertargets and then use this hdri to light additional 3D objects in the scene with it?
Hmmm interesting question. Not too sure but would love to try!
Welcome to new UE Plugin: "UEGaussianSplatting: 3D Gaussian Splatting Rendering Feature For UE"
Yo yo are u the dev for the new plugin ?
Yep, that's me!@@badxstudio
Send us an email or DM on Insta, we love to know how you developed it@@haikeye1425
What is the software used for gaussian splatter
Hello team, just a random question , can we create 3D models using guassian splatting ?
I really want to see more!
2-3 Videos of Gaussians coming your way already this coming week ❤️❤️
hey, i played around with that tool aswell in UE5 but i didnt get to the point to make the different parts of my gaussian splatting scene look seamless. you know what i am talking about? its dependet on where your viewport camera is. it seems as if certain parts of the gaussian splatting scene jump from one resolution to another. were you able to solve this? i looked it up in the documentation but i think i did something wrong. just for now i prefer the "sibr gaussian viewer" from the original github documentation.(just for viewing)
Hey are you talking about a seam (line) in your splat? If thats so check the documentation. You have to reimport it with a new console command to change the import method
So technically you could create a photorealistic 3d scene traditionally, then, pan the camera around virtually every single angle and take a video of that… then create a 3d Gaussian world based on that world.
Yep so technically you can do this in video games too :D
Nice! Could you use a negative volume or something to crop the unused parts?
Wait didn't we do that in this video? :D I think we did crop it for abit
@@badxstudio you cropped it, but I know for a mesh in ue you can crop or do a Boolean, I'm not sure with this point system if it can do same. Like shape a Boolean shape b to leave hole, might be a way to remove the objects in sky.
amazing stuff!!
Thank you
Hi guys and thanks for all the great content.
I was wondering to use this technique to test infrastructure assessment with drones (imagine capturing a full bridge by flying around and under it). Do you think this could work? And do you think that using a reference object for sizing it would be possible to take precise measures of parts of the structure?
Trying to understand if this technology could be useful to replace complex photogrammetry missions to some degree.
Funny enough we were just having a conversation about this between us ... You can bet that eventually these technologies will be used to perform precise measurements but we assume that it is not a 100% there yet when compared to expensive machines and detailed photogrammetry. However since the question was brought up, we are going to attempt to do a measurement test and see how accurate the results are. Stay tuned broski!
@@badxstudio great! 😍
So cool guys!
It's so amazinggg
The price of the plugin is worth it honestly
100% agreed. It got updated today again
Splat only showing in live mode, not in viewport. Is there a fix for this?
Hei! I have just downloaded Unreal and tryed to connect with Luma but it seems Unreal is 5.3 and plugin is for 5.2? Any help?
Broski this is not luma! This one is different ... its gaussians!
in theory you could make a scale model town and use that as your training
right?
Could you compare the unreal paid plugin and the luma free one ? Is one better than the other one ? Fps ? Yours videos are great btw
Thank you
I'm having pain because I have to wait for the next video
hahahahah brother it's cominggg very soon
Could you provide the link to download the photos to imput, like the piramid ? I love your channel!!
The photos are huge buddy, come on our discord and we figure out a way to give you the .ply file or the footage
ohh thats pretty good!@@badxstudio
I am on discord channel, how can you provide me the ply
Wow, just Wow.
Crazy. Right?
So now what we need is for DJI to add a feature to their drones to be able to define a target for calculating a flight path for taking splat maps
hahaha we tried Photogrammetry with our DJI Mini Pro 3 and the trick was to use the target tracker and keep taking photos. Similar thing can be applied here
Anything to use gaussian splatting in blender? I've seen a tweet from a guy who creates tutorial to make 3d gaussian splatting, so I just want to know if it exist / its possible to make. Any Response is appreciated.
Downloaded a plugin from github but unfortunately that didnt work for us! Im sure it will come soon 🤟🏼
@@badxstudio thanks for the response , and yea hopefully it Come Out sinon 👍
Dayum you bros have some uncanny valley qualities! : D
Only 1m4s in tho really liike it so far!
One Love!
Always forward, never ever backward!!
☀☀☀
💚💛❤
🙏🏿🙏🙏🏼
Cheers hahaha thanks a lot mate
Thank you for your enthusiastic self and being authentic : D
@@badxstudio
How is performance? Is this something we might see in a game ... or something that I might use to create a VR game that runs on Quest 3? Does this need as much VRAM as GSplats usually need?
Crazy fast and very high FPS. Probably not at this stage but for sure in future yes. It is very fast to view and render.
We are going to test it in VR in the coming week.
@@badxstudio amazing can't wait to see it!
It's amazing 🤩, can i use this in unity 🙄 ?
Hi just a short question can we achieve this with low vram gpu like rtx 3050 laptop edition with 4 gb vram the unreal engine works fine but it's the gaussian splatting I'm concerned about? Thankyou in advance
So you can't currently train your Gaussians with a GPU that holds less than 24gb vram ... BUT here is a solution!
Polycam just announced Gaussian Splatting on their website which is available for free.
You can test it there
@@badxstudio ohh great. Thank you so much for helping.
@@badxstudioyou can train with less than 24 gb i'm doing this with a mpbile rtx 3080 with 16 gb...up to 30000 interactions. With lower vram i Guess you should at least reach 7k interations where the 3dgs is saved as a first checkpoint.
Absolutely amazing. Link to the Unreal engine plugin please
www.unrealengine.com/marketplace/en-US/product/3d-gaussians-plugin
there u go!
do you think it would be possible to import a 360 video from a camera mounted on top of a Mavic3?
Should be! We havn't tried it tho!
@@badxstudio this would be awesome! Think of the amount of details you could reach
Hi. Were these locations shown as niagara particles? Wish you showed how the added lighting would’ve affected these scenes 😢
Yes they convert to niagara particles with this plugin! Don’t worry we are doing a video about it soon!
Have to try with 360 cameras to get the whole spots , then u got a really good map
I want to know if the size of the trained large-scale scene model is very large? A simple model I trained myself can reach several hundred MB
What about an indoor Nerf vs GSplat. ? I’m going buy this app next Friday and try with the Polycam Splats. And hopefully get around to the regular splat set up. Hopefully these can replace costly reshoot builds where we have to recreate a portion of a Set. And the background is just out of focus anyway
Can you test Polycam Splat Vs Original Splats?
Awesome channel BTW!
Next video is on polycam splats ;) There is a hint for ya!
You guys rock!! I don't have a very powerful machine so before when I tried importing NeRFs inside Unreal with another plugin it would crash. Do you think those Gaussian Splatting can load on a less powerful machine? Would you make a test file available to try?
Hey buddd!
@@badxstudio Super cool!! Gotta give this a shot!!
I'm curious, how heavy is a scene like this in terms of memory usage ?
Very intensive on VRAM.
@@ezbench_ Ah...that's a shame!
damn, thought you guys were gonna throw a third person character in there real quick over a plane so he can walk, and just make the plane transparent or something. That would've been real simple but cool. either way this is awesome
amazing
Thank you! Cheers!
Using this technology on security cameras.
The videos captured by them.
Exploring the environment to have a better look and understanding of the scene
Can you do a 360 video inside a city and use this as HDRI/sky dome for a game?
I tried doing photogrammetry with Google street view domes and it didn't work but I wonder if you could do it with this?
If you tried let us know :D
Ah yes, UEFN my home, can't wait to see what you guys do with this
Dude just a little teaser for ya ... UEFN is on our Roadmap HARD!! Just wait and see over the next few months
@@badxstudio sure !!! 😀
Ur tutorials are the best they are so immersive and on point just like your awesome work
HELLLOOO SIR! good to see you here
Here we go)
🙌🙌
very soon we won't be able to tell 3D from real life, wow!
It's getting crazy out here man!
You guys have to try gaussian splatting with a 360 Camera
Just imagine the Gaussian Splats Robots and self-driving cars would produce. They'd be able to "videogame" your life and generate maps for everywhere you go.
PLEASE do this real time!!! The end of problems with stream 3d content! 🤯🤯
How to create such point clouds?
You can either train your own using your PC or you can now use a website like Polycam that has their own Gaussian splats. .. we have an upcoming video dropping tomorrow night literally about that!
It always failed when I train my model, and shows cuda out of memory, anyone can help ? My GPU is NVIDIA RTXA6000 48G, and I input 999 pictures , they are 10G in total.
We need some Aeria 51 footage to analyse the place better :P
Reflections don't work with that plugin right? Imagine UE5 implementing GS themselves
Thank you for the video!
The tool is AWESOME ! I see you started from pictures or videos but would it be possible to start from google earth? :) Would it work with point cloud from construction site?
Check out Cesium real world 3d tiles
Possibly! We have another video on our channel for importing Google Maps into UE
@@badxstudio yeah I tried and commented about it, but this one the quality is much better ahah thanks for the reply you guys are dope af!
Big love brother
can you edit the niagara system?
Yes you can! We actually played around till we enabled shadows on the niagara now! But our experience with niagara is insufficient and therefore we havn't been able to fully utilise it!
Guys in the drone videos. I wonder if the drone goes 360 how it is going to be ?
I am doing an independent project, honestly many hate artificial intelligences, but I love them, without it I cannot carry out my project.
Also polycam has now added a feature for this.. but experimental and only 200 photos
Yes, and we are going to test that in the next video