Convert Ai generated 2D images to 3D models for use in Blender and Gravity Sketch.
Vložit
- čas přidán 28. 05. 2024
- This is an overview of how I use Ai generated images from Midjourney and Stable Diffusion to create 3D models that can be used for design reviews and refinement. I cover the 3D conversion process and how I manipulate and use that 3D data in Blender and Gravity sketch to help create 3D forms for further design maturation.
Use this to convert 2D images with depthmap estimation:
huggingface.co/spaces/shariqf...
Very cool workflow. I think the clean up in regards to the warped texture bits and the missing back and bottom are really tough bits to work out. Using depth maps like this is a fun technology and tool. Thanks for the vid!
pro tip: in blender if you set the fbx "Path Mode" to Copy and then click the little button next to it to embed textures, they should arrive properly in GS
AH thats what I was looking for. forgot the embed texture. Thanks!!
You're good at figuring this stuff out. You're on the forefront! Please keep sharing.
FINALLY somebody using gravity sketch!! thank you this is a game changer!!
My new favourite chanell - Looking forward to more Gravity Sketch♥ & AI smart optimisation solutions!
Awesome workflow! Thanks for sharing 👏
this is actually the FIRST ai workflow that I've seen how ai is intended to be used. we need more content like this. I just wish the model's geo was a lot better
The dev just needs better math implemented like smoothing edges like adobe does and a feathering option which doesnt seem too hard because its just vertex math basically. Like if the dev trained the ai to use those maths, it would be better
Just wait until it can do proper topology and fill unseen spaces. This one is a more "hacked" way of doing it, but already looking good. Not practical without decimation thou
@@pebegou wont happen.
@@jeffyboi6969 It not only will but it already does. There are AI retopology and shit that do tries to complete unseen spaces. The way this dude made it has zero input from AI to generate mesh. Also the ones we do have now is pretty bad for now, but with enough training one can even fuck yo mom.
5:00 Adding an up to scale character model or a face to the scene would make adding depth and fixing the size much easier. There are a bunch of free character models, just check if the character size is around 185 cm (or the equal of the character collision for default characters in the engine of your choice). Simply put the cut side of the helmet over the character's head so the eyes and eye holes meet and then mirror and scale what is needed. This way, you will use a single character model (size) for all of your projects and it would make it easier in the long run as well...
Agreed. I did this in gravity sketch so when I build the final geo I have that reference. its a scan of my head so I can pretend to wear it.. until I finally make it.. 😅
@@JamesWRobbins It is a good idea and an amazing method to save you some time! I sadly do not have a VR set, so I could not play around in the same way. That being said, I am also a lvl/ environment artist so my approach is naturally different. Scale how ever is one of the most important things regardless, something I had to learn the hard way once I started working in engine many moons ago XD
Its a fun workflow but i feel like the traditional modeling workflow might be quicker this over complicates the process of building simple geometry 🤔
But this also includes the generation of the idea. This illustrates how you can go from nothing, not even a concept image to a 3D model in about 15 minutes with the help of Ai and a little bit of manual work. In the future that will be the workflow.. This actually greatly simplifies the process. You don't need multiple views for reference, and then you have a 3D volume to build over with texture reference include on that volume. You would just have to do some retopology and cleanup. Depending on the final output needed, obviously the workflow for this will vary for Game asset to 3D print to full manufacturable part. But I'd love to see someone do a concept render and build a textured model in 15 minutes..
The ai idea generation is great my main qualm is with the 3d generation part from 2d to 3d, you still have to work hard to create the proper looking peices of geometry that the 2d to 3d software didn't get right enough for you to do retopology on. However This workflow could work for me if I use the 2d to 3d generated model as a base mesh that I'll sculpt and polish up in a sculpting software like zbrush and maybe remodel and replace pieces I don't like then combine it with the new sculpted mesh then finally retopology 🤔
Exactly. Its a good starting base because of the volume and texture detail
I am conflicted. The purity of hand modeling vs the reality of ai/ai involved worklows. But i got to agree. Minutes versus hours is invaluable. I don't wanna fight the technology advance. I want it to fight for me..👍
There's no way you can traditionally model this helmet faster than he did this process....
jeez 4:42 a select lasso for vertices omg never knew that thanks - and thanks for the rest too wow! amazing man!
Nice workflow, thanks!
Really interesting stuff, thx I'll surely try thos workflow
I’m a stylized 3D Artist and watching this makes me happy because now I can truly see my work flow is much faster and cleaner.
Wow! Very cool and interesting work!
Thank you, this was incredible.
This is great stuff! Thank you.
Keep it up. It’s awesome how you tried to implement new technologies. Don’t let negative comments hinder you. Subbed!
Thanks for the video mate! I am having trouble enabling the texture, when i click the top right drop down menu, i cant see the same options as you, i see less options (blender 3.5 btw). Is there a way I can make all these options show up or other way to turn the textures on? Thanks a lot!
You can take this and import into zbrush , add some guides and then z remesh a duplicate. This will clean up in quads but you'll lose your uvs. Now take the other duplicate and flip the uv map vertically , assign the texture from the model and divide in order to convert to polypaint , you will be able to store the diffuse map in vertex color info instead , bake that out of substance painter and work from there overlapping more maps and normals as well as perfect the geometry all this in blender. I like this workflow very cool
very useful for experimentation, for now
great stuff James!! Zoedepths is awesome!
Fantastic work. thank you.
awesome workflow!
This is amazing , never knew you could use this shortcut.
Still a proof of concept and a little work but not bad for 15 min of work
(5:21) You should do two things first:
a) use a plane or cube to "subtract" the fuzz on the back half of the imported model because there are a tonne of vertices you don't need that will bog down your performance, and a subtract will give you a nice tight edge.
b) scale it on the X (or Z) axis first because most of the proportions will be preserved, compared to "pulling" on the mask will cause small distortions you'll have to iron out later.
Yeah, I wasn't sure how much of the "Fuzz" would be used so that's why I use the cube to just hide it. Once its deleted I can't get it back so that's one way to "delete" it and still keep it in reserve. Not worried about performance with this mesh. Some might, so its a good idea.. Each model is off in some axis but it varies so I just pull where I feel its needed at first. its just a rough in so I'm really just going for a volume and texture with this for reference as a rebuild later. Its just a proof of concept at this point..
Adding a few primitives like a half sphere to the top of the helmet
As Blender beginner, i like seeing the keys u press.
Fantastic tutorial.
What a beautiful tutorial! haha Thanks man
great tutorial
thanks, quite helpful for abstract effect haha
Very nice!!
Thanks!
great video thanks!
very good one .thanks
With this I could make some pretty cool 3D printed cosplay stuff.
ГЕНИАЛЬНО!!! ЭТО ШЕДЕЕЕЕЕВР!!!
Cool and interesting
Awesome work - next step would be retopology back in Blender ie. trace the model to get something more final.
That's amazing
Great video, I was wondering is it possible to flip the 2d image in order to get the other side of the image so it less manual work that has to be done??
Hi James, I was just wondering how the results came out. Did you 3D print the helmet? Did the model come out good as expected? I'm curious.
nice work
Good tool! l am preparing for my new resin printer. Ai concept to 3d model to physical product.
Interesting workflow! Doesn't look very practical, but interesting nonetheless. I wonder if we'll still have physical monitors a few years from now, or if it's all in VR or AR.
I'm wondering what's ur prompt to the helmet, it looks so perfect!!
what a time to live...
Honestly, the forefront image at 0:00 should have been what the First Order helmet in Star Wars should have looked like. It's intimidating, looks like a natural progression of helmet design, and consistent with the Star Wars lore. Of course, minus the orange.
very coincidental i came across this. i just generated a mask on midjourney the other day and have been trying to figure out how to turn that into a 3d model
can we run zoedepth locally? and where do suggest i start for learning blender?
Good workflow concept, though I'd probably try to generate a front view and a side view of the object and then use a more traditional blender modeling approach to get a game ready object.
or do retopology of this and not have to guess at the volume. But depending on the final need there are many ways to make 3D models 😀. this is a POC/ experiment on possibilities.
Nice!
That helmet gives off helldivers 2 vibes
This will sound mad but is there a software I can add a pre made 2d animation and ai will convert to 3d or am I 10 years to early
Do you do these projects for other if requested.
Hi, good job! not possible fbx with texture ?
What you're missing is a better depth modes with multiple view angles and automation via the huggingface api.
in which year you are living!!!! you are way ahead of everyone
when i go to the link it displays a "Run Tim Error" and a long list, why is that?. but thanks for shedding light on this tool, absolutely magnificent.
Not sure, maybe a huggingfaces issue. its not my tool so I can't control anything with it..
New era.
4:32 I dont manege to delete the parts?? It delete only small parts of what is selected?? What did you do?
cool video✌
This might be ok for a quick concept but I don't think it's anywhere near as efficient as a normal workflow for creating something useable at the moment.
Making a depth map from an image like that is basically just going to give you a blobby mesh lacking any kind of detail. So to be useful for anything it will still need a lot of sculpting and retypo. Someone competent at scultping could create that rough blobby mesh faster with just some simple scultping.
On top of that the images from AI are going to have all light and shadow baked in because they are not albedo maps. That means they are noy ideal for textures anyway.
I think a better workflow is to just use AI to help generate ideas and then use those ideas as reference for your modeleling and texturing and just do it the normal way.
I'm sure at some point in the next few years AI will be able to generate 3D models as well as we can generate images now but until then I think traditional workflows are still the way to go imo.
Nice video anyway!
So basically you have to sculpt the helmet, ai only helps in projecting a texture, you dont need ai for that
What would be the best way to do this for a full body mid journey 2d character image to 3D model? I have a character that I love for a new animation, but only have one front-facing image of the character since I didn’t get the character design sheet while prompting in Midjourney. I can’t find any other answers to this online, so any help would be greatly appreciated 🙏🏾 thanks for this tutorial btw it was mad helpful
czcams.com/video/Kf9oSHRWdNg/video.html
10x mate!!
Though I wonder how do you select ALL the vertices in the Lasso, it seems like I delete and I still have vertices remaining in that area.
A tip for you dont delete the noise at all in adobe audition or any audio software you use that's make the voice looks weird
Wow do you do this for a living? What glasses are you using? Do you have any recommendation if just starting out wanting to make AR Games?
For example, I like TiltFive (aka CastAR) but I'm interested in more FPS and with more standard AR glasses (Google Glass/Hololens 2/Lenovo ThinkReality A3).
BTW, any thoughts on Luma Labs' Video to 3D API) , Blockade , Spline AI?
There are no good AR products released at this time, the best you'll get right now is the meta quest pro which is really built for VR not AR (this is what I assume he is using in this video), or the meta quest 3 releasing later this year.
"In the name of progress, and most importantly (!) -profit!; a pronounce The Death of Creativity ! "
what hardware do you have?
I'm not a 3D artist but I watched some of Nvidia's recent developer conference and they showed off some tech that automates this sort of process to some degree - taking 2d images into 3D space. I think it's called "Instant NeRF" or something like that. Cool to watch you do this. First time I've seen someone work in VR space. I've been generating some really novel forms with stable diffusion using 'sacred geometry' and Platonic solids, and while they're only like 80% coherent I feel like they could be a jumping off point to creating some fascinating 3D art or sculptures.
yeah its really nice to see my dream job getting automated...
@@METTI1986LA Is your dream to make cool 3D models and artwork? This is just a more sophisticated tool in your pipeline. There will need to be a human in the loop for a long time to come.
they just wanna complain instead of innovate@@parkerault2607
its crazy how you can now edit in VR
Hi, I looked at your app and unfortunately it is not functional at the moment? How can I help so I can try it out?
Not sure, maybe a huggingfaces issue. its not my tool so I can't control anything with it.
wow thank you so much dude
Topology has entered the chat..
Does it works with face images?
I have a question, How can I export as a render if the material doesn't show me? That's the negative thing...
After seeing the wireframe of this "3D Model" I really relieved as a 3D artist. With this mess of a wireframe, AI will not take over my job soon...
Sorry but beside some basic still renders a mesh with that topology can't be used for anything...
The sad part is that people with no knowledge of 3D art will use these horrible meshes in games and then wonder why things aren't working properly
Thank god somebody wise in comments, and the most sad thing -beginers will watch this dogshit process and will do more same million triangles helmets ))
This is sick. Had no idea this was possible
yes
6:40 horse helmet
:)
finally.
It's a hard-surface model. Could we use the method to generate 3D character model?
i would not sugest useing this for characters. cuse how it works details would be messy, like the hands the face over all. ect . and thats if you have a straght on A or T pose. if its a pose like crossed arms then youed pretty much have no arms or hands at all depending on the pose.
@@the_stray_cat I hope the next version of Midjourney can do that
Hry this is pretty good
Then, you have to do retopology. Using ZRemesher or alternative. Displacement modelling
Can you do this with a photo of a person?
I wonder if it can work nicely with my drawings to convert them in 3d
hi thx for the video its awesome could you save this blender 3d file dxf format thx again
this is great, though some retopology and edgeflow work would be needed after, this would shave off hours of work in the beginning
good for drafting and conceptualizing but i wont submit this quality to a client as final.
But wait, technically it'll work with non-ai concept arts too no? The possibilities are scary with this approach
Fuck yeah!
I want to learn
tytytytytyty!!!!!!!!!!!
hi, this is no longer available it seems... is there another one or a new upload?
i mean zoedepth
Not sure. Its working for me. I didn't create the app so I don't have any insight into its availability or control over it..
@@JamesWRobbins its working now, thank you
That's what your link shows "Runtime error"
Not sure, maybe a huggingfaces issue. its not my tool so I can't control anything with it.
3D generate AI to 3D Model to 3D Printing?
That's the plan. Working on that now..
Imagine it would be possible to generate the other views matching in AI so you have more reference images to generate the 3D model
that is kinda possible to do now
@@kamel3d how ? I did not figure that out yet
@@cekuhnen You can *sort of* do character turnarounds in Stable Diffusion, look into it.
@@j_shelby_damnwird yes I saw that with characters - but does that work with objects too?
@@cekuhnen I've been out of the loop of AI for a month or so (ages in AI-tech development pace) and as far as I know this was strictly for characters. Who knows? maybe you can get consistency in object turnarounds now!
much efficient is to design it from scratch all together in blender,
This is great. Specially for us beginners .
So its a blob with texture? Gotcha.
Awesome tool but it doesn't have form of material properties