Python __slots__ and object layout explained
Vložit
- čas přidán 8. 07. 2024
- Save memory using Python slots.
Python _slots_ are useful for saving memory when you have a large number of small objects whose attributes are known ahead of time. In this video we learn about __slots__, what slots actually are and how they work.
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
Slots docs: docs.python.org/3/reference/d...
Descriptor docs: docs.python.org/3/reference/d...
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, John Martin, Casey G, Pieter G, Krisztian M, Sigmanificient
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
0:43 Normal class
1:53 Slots class
2:44 Why use slots
4:21 Slots are descriptors
5:45 What are slots
8:32 Inheritance
9:14 No metaclass slots
9:24 Using a dict slot - Věda a technologie
“I’d like to thank me for sponsoring myself” 🤣🤣🤣🤣🤣
Internet, its like sitting on a commode writing on toilet paper, and then the papers talk to each other.
I would like to thank you a lot too.
It's not just the memory savings. Restricting the ability to add new members is a feature itself.
If I make a typo when assigning to an attribute, I want to get an error. I don't want it to be silently ignored, leaving me wondering why the value doesn't change.
That is really the main reason I always use slots. Memory savings are just a nice side-effect.
You're right, it is also a feature when you look at it that way!
You could maybe also catch these errors with the right IDE settings (i.e. let it warn you about assignment to attributes not declared in __init__)
I prefer mypy warnings for this purpose. Adding/editing __slots__ every time seems like extra typing
@@expurple Adding type annotations to everything is also extra typing.
If you use annotations you can add the following below the annotations of your instance variables instead of repeating their names: ___slots___ = tuple(__annotations__)
Finally, a great primer on slots!
Glad you enjoyed!
The way you pick examples is phenomenally efficient at highlighting limitations and advantages of whatever functionality you present.
Many thanks for the kind words!
this channel is literally a gem im soooooo glad i found you
5:51 "We need to enter the matrix"
Wouldn't this be more like leaving the matrix because you're peeking behind the curtain?
You're absolutely right!
“I’d like to thank me, for sponsoring myself.”
The reason to include __dict__ in __slots__ is because it makes it much easier to extend/modify the code, while _still_ giving you memory savings over the default.
Anything stored in the __dict__ incurs additional storage overhead not just because of the dictionary headers, but also for (a) storing the _key_, and (b) due to the additional empty nodes needed to make a hash table efficient --averaging about 3 pointers worth of storage per entry, while a slot variable always only takes one.
Therefore, any variable you can move into a slot represents a decrease in memory footprint, even if you can't move all of your variables into slots and still need __dict__.
Plus, current versions of python don't actually create the underlying dictionary object that goes in __dict__ until a non-slot variable on the object is written to, so depending on the usage pattern of your code, it could actually be _more_ memory efficient to leave a set of variables that only a small fraction of the objects of some class will use out of __slots__ and just rely on the automatic creation of a __dict__ in those cases.
Love it! As a C / C++ dev (in past) I can appreciate this. However as u mentioned we are rarely memory constrained. Thx !
There's so much information, my brain is frying. I'll be looking back at your videos constantly!🤯
I would also like to thank you for sponsoring yourself.
no useless words in the content, all are knowledge intensive explanation. perfect!
Incredible as always...!!!
Thanks!
I'm working as a data scientist for 3 years using python with no-CS background and the topic you cover is always interesting and mindblowing.
Thank you, your explanation was awesome! I've been doing Python for ages but never really bothered with __slots__. At least now, I know what they are, their trade-offs and what they're used for.
Great to hear!
Nice. Your content is unique and not found anywhere else. Keep up the good work!
Thanks so much!
Its crazy how all your videos are intresting and easy to follow!
Love your tutorials. Great work! Subscribed
Thanks, now i actually understand the previous video completely
I had been waiting on this! Really great you explain things well you are an explainer
Much appreciated!
I find not being able to add a member actually a very interesting feature. This is something I often don't want to do, but can happen by accident. Of course, mypy can catch that often.
Very clear explanation! Thank you very much :)
Love your videos keep it up!
Thanks, will do!
This is such a nice video! Thanks. Would love one on weakrefs or asyncio.
Best sponsorship in a vid I've seen
I like your sponsorship very much :)
great video!
I would like to see more c stuff , maybe even showing stuff on how python code translates to c code
That's the best sponsor spot ever!
I saw a comment on one of your videos saying "thanks Tony Hawk". Then I looked up Tony Hawk, and the comment suddenly made sense. Anyways, thanks for the videos.
Hahaha I'll take it as a compliment, cool dude. Thanks for watching!
Best python channel
Really interesting to me and useful. A debug through the C code would be very cool also.
I use __slots__ for a different reason. In the event that I write a new method and spell an attribute wrong, I find the bug immediately.
Excellent! Maybe one day I’ll use slots now 😅
This is GOLD
I appreciate the kind words!
this guy rocks
"I'd like to thank *me*, for sponsoring *myself*! Did you know that I am available for consulting, contracting, training and interview prep services?" AWESOME
Really great explanation (as always!) Maybe a video on weakref in the future?
Thanks! Probably will at some point but it's not a priority at this time. So many other interesting things to cover!
Really good video, as always. One request: can you do a video where you talk about making the Python classes in C? I use the CPython a lot to make modules (with only functions) in order to make faster codes. I was trying to make the Python classes directly from the C API but I have a lot of trouble. Thanks!
I will probably make a Cython video at some point, not sure about a pure C api video though. Thanks for the idea!
Great
This video format should be tried more at schools and universities.
Yessir!
One tradeoff of slots that's not immediately apparent is that it can break multiple inheritance: A subclass can only have one parent class that defines nonempty slots. Presumably it has something to do with how parent and child slots are concatenated? Like in C++, for single inheritance you just tack on extra fields, and bam, the address of the child type is also valid as an address to the parent. But with multiple parents you need special casting operators. But since Python is using descriptors for access, I still don't quite get why that's an issue.
I feel like I can use this as using C Struct, where we already know its member ahead of time
what’s the recursive “getsize” function you use? great work as always!
Wow that was very komakai. So the primary use case for using slots is when you have a lot of properties or when you're creating a lot of the objects?
I haven’t used slots to save memory, but I have used to them to avoid a certain class of errors.
I have some classes with writable properties. In particular, I had a property named “dash”. One day, I was writing code like
ctx.dashes = «dash-settings»
and I was wondering why the dash settings were not being correctly set. It took a while to realize I had written “dashes” instead of “dash”, and so instead of updating the correct property, Python had silently created an additional, useless, attribute!
From that point, I have been careful to add slots definitions to such classes.
However, if you then want to have weak references to instances of such classes, you must remember to add the name weakref to the slots list. This is where Python stores some kind of backpointer for keeping track of weak refs.
This for micro-controllers is very interesting
what i miss from this video, is that, is the slots reserved for the variables you declare in "__init__" only or they go for the function definitions inside class themselves as well?
I have a question: in my case I use fastapi with Pydantic classes for models, does using "__slots__" make any sense here? Or it not for my case?
I remember I watched a talk from Richard Hettinger mentioning the use of __slots__ but he could not go into the details probably due to time constraints. Although I am not likely to use __slots__, thank you for making this video!
I learn something new today!
One question though: when we define a __dict__ in __slots__, does attribute assignments work the same as if __slots__ is not used? Is this __dict__ in the __slots__ of the superclass still a dictionary? And does Python uses this __dict__ when we assign new attributes dynamically to instances, if possible?
Thank you!
Yes, it will store variable not in slot into __dict__, also your question can be answered just by opening python repl yourself, play with it.
Can you create validator descriptors for classes with slots? It seems like setattr() and getattr() don't work the same.
What about the case when we will have a list object as a slot? Does that make sens?
0:30 oh yeah good ol' me ;)
@0:34 #LOVEIT!! 🤣🤣🤣🤣🤣🤣👏👏👏👏
Are there any nice tools that can visualize the memory layout of python objects just like your diagrams?
That was like super awesome!
Glad you liked it!
what about '__dict__' AND '__weakref__' in __slots__? will the class be slotted WITH dictionaries and weakrefs?
Nice video! Now I finally understand slots in Python
How can I get the sizes of ssize_t and ptr in Python?
It would be nice to know where you search for information to prepare material for videos
Just curious, is there any reason why you are using .__dict__ instead of vars()?
Hey, do you know where I might be able to find a similar colour scheme for VS Code?
looks like monokai
I've spent days trying to understand python code, which at runtime assigned attributes to objects, whose classes were also determined at runtime.
Anything can be anything.
This is both the gift and the curse of the language. Such dynamic abilities allow for so many possibilities, but many of those possibilities are terrible ;)
So the size of a slot instance without recursion may be much larger then that of a normal instance without recursion.
Are classes with slots faster? One fewer dictionary lookup must count for something, but is it noticeable in reasonable use cases?
No, it doesn’t give noticeable speed difference, only memory
yes, one lookup is not that much faster but the more you have the more you save. And why not if you can.
@@leobozkir5425 I would say to never worry about optimisation at first, but if software demands then you can.
Does the benchmark at 4:10 not answer that question? It seems to be 10-30% faster, but this probably not really relevant if your have performance issues with python, there are other tricks that make a much bigger difference.
I'm going to try to summarize what I learned in 1 paragraph, as I find it a helpful exercise, and I want to feed the algorithm :)
When python classes store data, is uses a hashmap (__dict__). If you want to use an array of pointers instead, you accomplish this by specifying "__slots__". This has clear benefits if you care about the differences between these two data structures.
That's pretty much it!
This would be useful in the case of Cloud Functions where you have to pay for memory used by your program.
8:46 => could you explain Instance memory map for inherited class B(A), where A is slotted class but B is not.
James -- what point were you making in the title, listing dunder-slots and slots separately? (BTW, nice choice of topic, and delivered satisfyingly succinctly.)
The dunder slots are just the _extra_ slots a class defines.
@@mCoding Thanks for the reply, but ...well on the one hand, yes obviously slots are data that's 'extra" to whatever per-instance data Python needs on each object to manage it. But such housekeeping implementation data doesn't quite seem appropriate to refer to as "slots" in the context of user-useful data slots. Is your notion captured in your "graphic" at 7:27, where a label "each is a slot" refers to ob_refcnt, ob_type, x, y, z? In what sense are ob_refcnt and ob_type "slots"? Are you using "slots" here as a generic synonym for "sequentially-stored same-sized data members" (here, on a B object instance)? Or is there some realm (maybe in the C implementation) in which there's an actual array of these same-sized items named literally "slots"?
So this makes classes more work like they do in C++,C# etc.Where you got predefined fields(instance variables) and you declare them static if you want them to be Class only.
How does changing a slots variable from an object instance work?
I would assume that the value of that variable would change across all instances of that class. but it doesn't.
can you explain why?
it's just an instance member, changing it doesn't affect the class or any other instances of that class.
at 1:54 just to add some info to the viewers. This is monkey patching. Although usually we use it to change or add some method instead of just an attribute. Very pythonic!
Good video, but it doesn't really explain why slotted 7-attribute object is 5 times less memory demanding than the non-slotted one. In both cases we still need to store the data somewhere (either slot or a __dict__). Why storing data in __dict__ is so much more expensive? Where's all the additional memory consumption coming from? Genuine thank to anyone who takes time to explain me.
Using a dict is just that much more expensive than array like storage.
As for why that is, consider the extra memory a hash table (dict) needs to hold in order to reduce the chances of a hash collision.
Good point, thanks. Apparently dicts saves hash codes and pointers to key+value tuple.
Don't forget you hit the like button odd number of times in total, including all of your other visits to this video under the same CZcams account.😇
How do you calculate the recursive size of an object? Would like to see the `getsize()` function :)
See the source code in the github!
Well, always have been a philosophical way of how to measure ones dict size correctly...
1. Is there a non-hacky way add type annotations to slotted attributes?
2. On 2:36 you say you can still add class variables to the class as usual. Can we go one level deeper and restrict that too ...with slots? ...without slots?
1. You should add type annotations wherever you would normally initialize the variable (typically in init). 2. That would involve setting the slots on the metaclass, which gives an error if you make slots nonempty, and even if you make it empty it for some undocumented reason still doesn't prevent the class object from getting a dict, so it seems like you cannot prevent a class object from getting a dict.
@@mCoding I see, thanks for clarifying it! :)
discord gang 😎
Will watch later Cause didnt have Time now
Kudos on being full of yourself. What do you ask per hour of your time?
"I'd like to thank myself..."🤣🤣🤣🤣
Can you do a video on weakrefs? 🙏
It seems like this slots thing should have been the default behavior of a class.... The ability to add attributes dynamically on-the-go is not the way most uses of classes are, even in python. I wonder why that's not the default, rather than the other way around, especially when it also saves on memory and although that's the standard case it is much less known.
But those __slots__ pointers have to point to something - even if it's memory managed down at the C level (heap, presumably)?
Yes, nearly all Python objects are heap-allocated, so even with slots there is still one level of indirection. However, this is still better than the two levels of indirection used when a dict is used. Good catch!
I think this topic needs a bit of update regarding Python 3.11 objects with lazy dicts.
19th :D, yeah very specific ik
Confirmed!
lmao 1 year ago, still come back to watch your content as it is very helpful
Keep going!
U consult also on Dart?
Unfortunately no, I don't know Dart and therefore do not consult on Dart.
Discord gang
3rd
Hit that like button 5 times
0:30 just a moment i was expecting Snoop Dogg speech:
czcams.com/video/wGRF3GQ4Wdk/video.html
1. I use MATLAB for work - and MATLAB doesn't allow assigning variables to classes on runtime by default. Does that mean it uses slots? (Is that the same for C++?)
2. So the class with slots stores the pointers to the data as an array? Where is the actual data stored?
Most languages, especially statically-typed languages, use what Python calls slots by default, and hence there is no specific word for it in those languages. It just means pre-allocating some memory on an object for a specific variable, which is the simplest and arguably most efficient way to store attributes. What Python does by default would be laughable in most other languages as it is equivalent to proxying all attribute lookups and assignments through a dictionary member, which is wildly inefficient. But, of course, nobody uses Python for its runtime performance.
Awesome video and great examples, but your face is too distracting 👀 Ye too handsome
Help me
Too many commas in that title
Lol ty fixed
A hacker give this code to you
def password(str_input, str_key): -> int
Password_num = 0
str_key_len=len(str_key)
str_input_len=len(str_input)
for i in range(str_input_len):
Password_num += str_key.index(str_input[i])*str_key_len**(str_input_len-i-1)
return Password_num
This function can take a string and output the password needed to protect the string
Puzzle: Make a inverted function where argument are password and str_key
Input:
Line 1: password
Line 2: str_key
Output: A string
Constrains: str_key cannot have duplicate letter and must have at least character within the str_input
HELP
I love python but OOP in python is ugly
Why is it ugly?
@@mishikookropiridze Verbose syntax:
class Foo:
def __init__(self, a, b):
self.a = a
self.b = b
...
If I could make a suggestion, your very well made videos might be even easier to follow if you used more descriptive names.
Instead of calling a class "A" and and it's instance "a", calling them ClassA and inst_a could make it easier to follow as you speak. Just an idea.
That's not the Pep8 standard though. Camel space for classes and underscore lower case for instances is the norm. And Hungarian notation as a naming convention is rather a pain
@@QuantumHistorian I'm confused. "ClassA" is camel case, and "inst_a" is lower case with underscore
What am I'm missing? 🤔
@@virtualraider A and a also has that naming convention. What you've added is a prefix describing the type of the object, which is known as hungarian notation. That's rarely done these days, especially in languages that dynamically typed
@@QuantumHistorian oh, I see. Just to clarify, I'm not proposing that real code should be named like that, only for these videos.
Usually the explanation runs fast and he always has to clarify that "little a" is the instance and "big A" the class and so on, so this would make it explicit.
this is amazing feature with horrible notation. Lets think about a decorator for that...
Did not work, so I googled it. Works if changed to:
Class A(object):
Hi, it looks like you are using Python 2, which reached its end of life years ago. Please upgrade to Python 3 in order to follow any of the videos in this channel.
@@mCoding version 3.0 broke python whose development has been frozen. Please refer to the language you are using as Cobra 1.x.
actually, it can be an str itself (when you have only one attribute). check this out on documentation docs.python.org/3/reference/datamodel.html#object.__slots__
Transcripted from the reference: "This class variable can be assigned a string, iterable, or sequence of strings with variable names used by instances"