Hey jon , me and my father (an electrical engineer and i am in IT) watch your videos all the time. He loves your content and reminisces on his long history in EE. If you're ever in Toronto let us know
One memristor replace a digital macro component made of: 1) An A/D converter 2) An 8 or 12 bit Flash memory 3) A D/A converter. The applications of memristor arrays as computing memory really depends on the voltage retention qualities of the device. Thanks for the video, Anthony
I really appreciate this. I'd always hear about Memristors online, but not quite get what they actually were. This helps a tonne, and I am thankful for it. The concept of analog technology in a surprising field does remind me of something - have you ever been interested in what the life and death of the Scanimate machines were? Those analog CGI machines that produced a lot of CGI graphics for TV and film from the late 60s to the mid-80s. People seem to completely forget they existed when it comes to discussing CGI, and I'd argue they ended up being an essential and important stepping stone for the eventual incredible success of digital CGI. The way the machines work is both incredibly familiar to how digital CGI worked, but also completely alien. But it's fine if this sort of thing doesn't interest you. I'm really happy I got to watch this video, it taught me a lot in something I was curious of, but was too afraid to ask. Keep it up, these videos are a highlight for my day, and do genuinely cheer me up.
I remember when they were a big deal around a decade ago. For my research, they've been used in memory for a while. But they never really worked out as a universal gate. Now there's been several AI accelerator chips that have been posting insane numbers. I immediately thought of memristors. It's like they were almost explicitly made for matrix multiplication.
You had me at " these 'bad-boys' ". My chem profs back at university pointed that out - everything was a 'bad-boy' to me. Glassware, reactants, terms in equations. Bad-boys all. It was used so frequently that a few of them started using it too even. Anyways, sick vid, sick channel. Liked, subd.
During my undergrad my research orientator said that this would be very useful in the future. That was 2018. Here we are five years later and slowly but surely progress is being made. I doubt it will ever reach peak commercial application, mainly due to manufactoring constraints, but sure is a cool niche piece of tech.
Even if it doesn't reach mainstream mass-market use, I can see how the technology may end up being a key-tech in the future: having a semiconductor that can take on more than 2(3) states at the same time, and also include a memory-function within it, may be crucial to develop General Artificial Intelligence.
Current computation based neural networks are just stupidly inefficient, to determine whether an artificial neuron should turn on you have to read a bunch of values from memory, which is not in the same chip so you gotta go through a bunch of memory management crap, wasting a lot of power and time, then you gotta do the math which is basically unnecessary because neurons don't need precise numbers anyway. Then you take your result, put it back out into memory, so it could then be used by another round of simulation of another neuron down the line. Where as for a real neuron, this whole process is basically just baked into the wire, you send information into the wire, it automatically gets transformed and transferred over to the next neuron, there's no unnecessary nonsense like math or memory. A big reason why current deep learning based AI is so stupidly inefficient, but if we could somehow have hardware that does the same thing as neurons, without using math or external memory, it would instantly be orders of magnitudes more efficient.
Ok, it's not THAT inefficient. You can divide up layers on different GPUs, so they don't have to fetch weights from off their own VRAM. The most efficient fused kernels, like FlashAttention 2, try to do as much as possible with on-chip cache while minimizing calls to VRAM. But still, a crossbar memristor array would be like 10^6 times more efficient.
Give it time. Niche applications, if cheap enough. But large computing arrays - er not if it involves too many subtractions of large numbers. Remember the Intel 486 multiply section?
Qinghua University in China built a memristor chip several months ago, and conducted test in AI, and found to be much better than the regular approach. I personally think that by building a Field Trainable Memristor Array (FTMA), similar to the Field Programmable Gate Array, one can achieve AI function very similar to our brain. There is no need to worry about the imprecise nature of the memristor as our brain is also imprecise in nature. Jeff Hawkins of Numemta explained it very well, and has come with highly redundant models similar to our brain that can overcome this shortcoming. Using this approach, AGI can be implemented very quickly, with efficiency and cost very similar to our biological brain. Yes, AGI is finally coming faster than we think possible.
This is cool for me because I used to be in a group BEAM memristor group where guys were working on making them at home! Recently I've learned more about electronics and now there's all this legit research on the topic and it's great stuff.
I used to be part of a BEAM robotics group too, and before it died out, one of the last major discussions was the possible applications of memristors, and there were a few experimenters trying to make them. I think all of us had seen the Nile Steiner video and his home made memristors. I did 2 experimental designs, and did a full concept for a third, that I never tried out. First experiment was an absolute failure. Tried using silver paint to create the electrodes through a pinhole in an insulator over a Copper Sulphide layer over copper. The Copper Sulphide was way too porous for the silver pain and it just soaked in and shorted. For the second design, I bought 0.6 mm diameter aluminum balls from McMaster Carr to act as point contact electrodes. I created a sandwich between two PC boards, with the bottom layer masked and the copper pads reacted with sulfur to create the Coper Sulphide electrodes. The top layer reused the silver paint as a bonding agent to secure the aluminum balls to the top board. I screwed the boards together, but imperfections meant most of the 16 possible memristors on the board were either not under full contact, or were crushed by the force of the screws joining the boards. I had two of 16 memristors show hysteresis for about 2-3 test cycles before breaking down... a small success, experimentally, but ultimately, another production failure. My third design, which I wanted to try, but then the first discrete chip based memristors hit the market (though costly). I never followed through, though I've considered going back to it. The design used a thick "sturdy" PC board as a base, and again, silver paint would be used to bond tiny Aluminum balls to Copper pads. these pads would attach to a 20 pin header soldered to the board. a spacer board and a "finger" board would be made from 0.5 mm PC board material. the spacer would support the finger board 0.5 mm above the base PCB, and the fingers would look like a comb with header pins on one end, and thin fingers tipped with a copper pad on the other end. Those pad tips would be reacted with Sulfur to create Copper Sulphide electrodes that would have only the force of flexing an 0.5 mm PCB by 0.1 mm. My hope is the much lighter force would not crush the fragile Copper Sulphide layer, and allow memristance to be observed. By having two rows of 20-pin headers, the part would have the footprint of a 40-pin DIP, be usable on breadboards, and offer 20 memristors in the space of a 1980s era CPU. That was the plan... I didn't learn KiCAD for another 4 years, and since then, KNOWM has released even more memristor options for discrete experimentation. Still spendy, but not as bad as the old Bio Inspired versions from several years ago. I'm glad this video is covering the more analog capabilities of memristors. So many companies just looked at it as if it were just RAM 2.
I wonder how you learned all that. You make it sound so simple. But if I go out searching for all this I might just end up scratching my head. Amazing videos and amazing work.❤
I wonder if the defectivity may be a smaller problem than it looks on the surface. First, current research into quantized models has shown good-enough performance at inference time all the way down to 4 bits, which would at least seem to suggest a good bit of tolerance for poor precision as long as the overall design of the system takes it into account.
Nice video 🙂, I had this in my mind for research in the final year of my college before HP's patent and research paper came out. Application looked like first principles approach during my college education.
There were two small start ups working on this in the US in the past few years but one of them went under like a year and a half ago. Definitely promising just needs the right team and right market concept. Edge computing the demand just isn't there yet and it's a lot of infrastructure to build up, I think it'd be smart to start with gaming accelerators - make a PCIe card that has a series of chips for accelerating vocal intonation, NPC attitude, body language, facial expression/animation to line up with speech, all these little things to really make AI use in gaming pop. You probably couldn't load up an LLM, but you could locally turn the flat speech sent by an LLM into dynamic responsive NPC behavior. like a "Narrative Processing Unit" instead of the "GPU" the "NPU" or some crap like that. It's a concrete goal, you could turn out a decent, usable product and partner with a couple modder communities and big titles to get enough buy in to set up widespread consumer sales, then like a year of work and testing by the team and use the profits from the first run to expand out into the wide variety of areas this technology could be awesome for.
Hi, not really, memristors (curent Halfnium oxide ones) are very noisy, and have huge failure rates, plus they actually consume a lot of current to be programmed. So they are ideal for ROM, or Inference only accelerators, but for now it is not really though of as cache (PCM or other type of devices are tho).
@@theoballet1519 Of course they aren't good for it now, they're pretty new and underdeveloped compared to SRAM cells. The properties of an ideal memristor would still make them interesting potential alternatives should one ever be developed in a process node compatible with a processor manufacture node.
I can't even have words how much i love your videos. The information and the clarity of presenting the information are amazing. Thank you very much thank you a million times
I'm not sure if it works the way I am thinking of it or not. What if you have a material that would change its resistance if you sent a certain voltage or current through it, and it was resetable back to the lower bias state. Then you would have basically flash ram made out of something else. Would it last longer? Could it be as fast a sram?
Leon Chua is a legend in semiconductor circles (and for good reason). However, believe it or not, his daughter is even more famous. That would be Amy Chua. She is the author of “Tiger Mom” (a parent memoir) and the very important (if not well known) book “World On Fire”. She is the author (or co-author) of at least seven books.
Sounds like Tiger Mom didn't quite measure up to her Dad. He invented a world-changing technology that no one really believed could be true. Amy figured out a way to sell books that state the obvious (discipline your kid, democracy isn't always good, some groups make more money than others).
@@joeyp1927 What the father thought of the daughter isn't quite clear. He was opposed to her applying to Harvard (because she would have to leave home) but boasted about her being admitted to Harvard. She has written a number of books of which "Tiger Mom" is just one.
Intel made an analog NN chip in 1990 based on flash memory cells (aka isolated gate FET). It wasn't a commercial success but I believe it's likely the best approach. Flash cells can be made in any normal CMOS process with very few extra steps. Analog NN really are many orders of magnitude more efficient because you're not shuffling data through your limited number of compute units. Digital NN will also benefit from having compute local to every memory row, but the chip real estate would get enormous.
this is so so great videos, thank you so much! i make my own DIY DSSC now i wonder if i can DIY a simple AI chip with memristor at home with DSSC materials!
I am studying electronics (not on engineer level), and while many of the concepts this channel covers fly way above my head, it is fascinating how relatively short the history of electronics is and how big of an impact it has made on our society.
I wonder how much training with dropout (or a similar process modelled on common memristor defects) would help with models running on CMOS chips which may have a few faulty memristors?
I attempt to fabricate memresistors at my college clean room for a research project. We used an TiO2 sandwiched between aluminum. The sol-gel was a problem for us
memristors have also been tested as SSD and ram but i forget what happened but to say its forgotten not really i think they might use them in IC already as well plus circuit sim has them in there to play with. But use in Analogue AI processing yeah that is new.
I just learn so much from this channel. I definitely understand more about the whole chip business. I’m thinking I could save a struggling chip business….😂😂🍻
I love memristors and was really sad to see them go the way of the dodo in storage. I also find the possible return of analogue computing in ML fascinating, so to see both combined makes me unreasonably happy 😊
The memristor is a non-linear device with two terminals. You have Diodes and Thyristors that fill this role. Or you can use a transistor and tie the base to the source or drain to form a two terminal device that has non-linear IV curve. No need for memristor at all.
This was a tough topic and concept to understand at least for me. But you sir described this in great logical chronological detail. That even I could understand it. Basically the flow of electron get sifted through a “movable” variable resistor. Which multiplies it and then gathers those values and adds then giving you an output. Did I get that right I hope. I don’t know why it muliples it maybe I said but I forgot. And why add. What’s the output for. And also how is this memory hwo is it a seeding something that happened in the past to use now. And fyi I thought at the beginning it would work more like the resistance changes depending on the previous resistance needed, and I guess when a new current or voltage would come it change again to that past resistance value leading it to idk change on the fly. Maybe that’s nano tech sci fi but that was just what I though going in. But coming out I now think differently. Again thanks for the breakdown and detailed video thank you and I hope you do more and carry on
This may possibly be useful, but it is definitely extremely interesting! I wonder how long the memory cell will retain its value before it needs to be refreshed.
This is one of those things like oleds where it needs a killer app to receive the research to make it viable to be worth that research. oleds languished for decades as a curiosity until phone lcd backlight display power started being a significant limiting factor in battery life. oleds still haven't even come close to supplanting lcds for larger displays but the chemistry has been done to provide stable organic leds. I think the concept of a memristor array is going to have a tough fight with in-house AI implementations. The exact nature of the nonlinearity and process variation will put pretty strict requirements on the ADC arrays for resolution, and I think that itself makes for a challenging computational task. If you need a 24 bit ADC to get useful information out of a memristor array when a TPU is using a custom 7 bit float in it's MACs, there's a significant digital tradeoff between a memristor array and "conventional" digital AI training even excluding the analog performance improvements. Also, is it just me or should there be a DAC on the other side of the memristor input line? DACs are quick and it saves a step in the MAC process. Also also, This is the same Chua who invented (ish) the Chua circuit, which is cool.
What kind of fab capabilities will be required to produce the first commercial memristors? Will it be limited to TSMC or will some of the Mainland fabs be able to do this kind of research and production?
I sat in a talk given by HP about 15 years ago on memristors. . A room full of smarty pants’ from everywhere. After an hour I was pretty sure everyone knew less than when they walked in.
Another interesting application of memristors could be storing multiple bits in a single cell, greatly increasing the potential density. This would work by the senising voltage being run through an ADC to determine the bit value of the cell. 4, 8, or even16 bit values in a single cell should be achievable. A further benefit is that memristors can with further development could also become viable as non-volatile memory with potentially far higher speed than current nand-flash technology.
Multiple bits in a cell? They do it now, the limit is precision. SSD modules deal with that by swapping-in new blocks as they near unreliability/ageing. (a computer on a chip no less) Analogue may be fast, but precision is ALL. Or a moving feast usually. But yes non-volatile, competing with NRAM.
Memristor and nand flash are completely and fundamentally different in basically every aspect. A nand or nor flash is a so-called charge-trap design that stores electrons, ie. a charge. The amount of charge it can store, and by extent also the number of bits each cell can store is determined mostly by material purity. Even the most optimistic researchers do not expect to nand/nor technology to be able to achieve beyond 8 bit cells. A memristor doesn't store a charge. It changes resistance. To read it you send a sensing voltage through. This can easily be amplified and run through a flash ADC, limiting only the number of bits by the required speed. Typical 8-12bit flash ADCs can operate in the 10GHz range as used in oscilloscopes. The other main difference is that memristors can be read/written one cell at a time whereas nand/nor flash can typically only be written in 4-16KiB pages. Making both potential access time and bandwidth speed in a whole different league from nand/nor flash technology.
Sorry to be pedantic, but Von Neumann is to be pronounced the German way, with V sounding as F. For reference to viewers: at 14:41 BE is Bottom Electrode, TMO is Transition Metal Oxide, TE=Top Electrode and OEL=Oxygen Exchange Layer.
if they can figure out how to combine this with on chip forward-forward training (much simpler than back-prop but still good learning ability), it would be a huge step in using AI in a way that is not so centralized and does not need network connection, while being able to learn as it goes
You reminded me of HP's "The Machine" a memristor based processing system. very exciting until you realise it was just a marketing tool to get more money coming in via shares. It quietly disappeared a couple of years after the hoopla surrounding its release.
I wonder why nobody mentions the first application of memristors in 1939 by Hewlett in his Version of a Wien Bridge Generator. This dynamic memory- resistor was in use until the 1980s and is known as historical circuit.
Oh wow. This brings back memories. Please pardon the pun. lol But, no, I remember this debate. I remember being surprised that it got so much traction. It was an epic nerd fight, but I remember at the time concluding that the fight wasn't about anything at all important or interesting. To me, anyway. Thanks. A lot of your videos are great nostalgia hits. I didn't get to Asia (Tokyo, mostly) until the early aughts. But I was in Europe a lot before that, and worked for and with a lot of companies that came out of Mountain View and Palo Alto and other places. Software. As I age further into the back-half of my century, it's fascinating to see history's consideration of my past environment. An experience that latter-middle-aged people surely experience far more intensely today than their forebears, because everything's moving so fast. Then again, everything's relative, I suppose. These brains are curiously flexible that way.
the nature of consciousness will never be resolved but when we can replicate a brains function in wires we can arise many questions and make some of the oldest simpler and some of the oldest worse
Hey partner you okay. Because you sound a little off. It is appreciated that you post the video on schedule like normal. But if you need to take a break. I think we can all agree that that would be okay..
This is HUGE! 🤓 So, in the near future we'll literally have pre trained AI models running on our devices not just on one giant power hungry, super-computer that the whole world shares! (with possibility to change the weights later on, for example to update your AI assistant)
as long as we're looking at computing in memory for AI, what about memristor-adjacent ideas like Mythic AI using flash memory cells as the variable resistors in a similar array.
Hey, so... If you could get a memristor to quickly react to a high or low current and match it, wouldn't that just make... Ram SSDs? Like, actually. Could also be really good for CPU/GPU cache. Edit #1 : I know memristor are, by nature, already RAM - but _analog_ RAM. Doing what I suggested could make it binary, thus much more suited for non volatile RAM and cache. Like, could you imagine having a power outage, and when your PC turns back on everything's exactly as it were before it crashed? Edit #3 : Actually, why limit it to "0" And "1"? By reading multiple levels (Let's say 0 to 3) you could have incredibly dense memory/cache (in this example case, 2*2*2*2 so 16 bits of data) similarly to how high density SSDs already do with different charge levels. Honestly, the memristor could probably fix the "cache apocalypse" of TSMC's 3N and below nodes since they're a single transistor instead of many, even if they might be a little slower at first.
What i want to say - memristor is magnetic, flash uses electric field requires voltage pump migh be used to hold analog data Holtek once manufactured analog flash memory to record voice messages
7 bit flash cells have been prototyped - that's 128 levels - more than enough for limited precision ai netowrks - could a custom flash cell biasing network allow for a similar AI circuit?
If memristors depend so much in the fabrication yield, then why don't they implement a FinFET approach, which already looks similar to the memristors presented in the video, in order to effectively increase the probability that a single fin is correctly manufactured? Does this make sense?
Neural network of human brain is not analog. It is basically digital: A neuron can be activated or deactivated like a digital switch, not in-between. But it has variable number of input signal threshold to activate the neuron. kind of summing OP-amp/comparator. Resulting circuit work like analog circuit. Digital CMOS circuit can behave similarly. A array of switch coupled with digital memory cell also can work like summing OP amp. Not computing in digital binary number, but summing number of active inputs in analog way, though inefficient and low precision.
The concept is wonderful, in concept. Reality has to be factored in. And the biggest factor is what limits all analogue computing. Precision. Particularly devices that have non-linear properties. The Intel 486 multiply section had a small digital error. Only detectable when you start to compound arithmetic cycles, like subtracting large numbers to get relevant small differences. Accuracy is then ALL. Analogue may be faster - "but" once it gets complex ! Think human mental arithmetic. Large numbers. Any operation. How good are you?
@@giusdbg the techniques that allow digital to run fast will allow memristors to run fast enough to compete with an algorithm favourably. Horses for courses. Precision and complexity will be the digital domain. And multiplying with a memristor is basically "parameter 1 set R, then pass current for parameter 2". easy IMNSHO, but slower. The matrix addressing is by definition digital. So in reality hybrid. As for logarithms, diode have been made with such characteristics.
Stacking these would be insane, as memristors are incredibly power efficient. You could have something orders of magnitude more powerful than the brain in a deck of cards…
Are defects really that problematic, because values in neural network applications are kind of redundant, a few changed values here and there won't change outcome much.
Neural networks do a lot of math, particularly a lot of matrix * vector operations. In a regular, digital computer, this would be done sequentially, multiplication after multiplication, which can take "long". Even on parallel hardware this could take multiple cycles (I think). Memristor arrays could simplify the process by just having the input vector as a set of voltages on the top lines and reading the result of the matrix multiplication in the bottom lines. All memristors are working in parallel, making the process almost instantaneous.
@@zokalyx - TY. Is it the same idea as that of physical (hardware) neural networks that I've seen somehwere that is a (potential, under research) alternative to virtual (software) ones, which are the ones being used now in AIs?
@@LuisAldamiz Yeah, in some papers, they named it 'Hardware' NN to differentiate against regular 'software' ANN running on regular digital ALUs. The physical properties of the material (in this case, electrical conductance) and the linear law (ohm's law and Kirchoff's current law) is used here. For example, to do 4x2, 0.4V multiply by 2 microSiemens = 0.8microAmps. 0.8microAmps as an analog value, can be converted into digital 8. 4x2 = 8! Kirchoff's current law is just to add up more matrix elements
@@timng9104 TY for the reply. However I recall that one characteristic of these brain-like (under research) physical neural networks was also to imitate the way brains work, which is "slow", "low energy" and "chaotic" (as opposed to the heavily centralized CPU architectures). This is, I understand, not necessarily what a "simple" shift from transistor (CPU & RAM) to memristor architecture but retaining the current mainline "neural network" paradigm would do, right? IDK, probably there are several parallel and somewhat related lines of research here, all cutting-edge and thus a bit hard to understand. If they succeed, they may be behind a new AI and overall computing revolution (for the good and for the bad). Or it may not work at all, like the alleged graphene revolution and such.
Non-linear two-pole devices already exist, in the form of diodes (change when you overcome the forward voltage) and ... fuses (change permanently if you pass a too high current through it). I'd think the fuse better reflects the sand analogy and the memristor sandwich looks remarkably like a diode.
Hey jon , me and my father (an electrical engineer and i am in IT) watch your videos all the time. He loves your content and reminisces on his long history in EE. If you're ever in Toronto let us know
He can't he is actually an advanced AI from the future.
From a fellow Torontonian, I'd be glad to invite your father, you, and Jon, all on my account for dinner.
Gatdamn Canadians are nice! @@bee1707
@@bee1707as another fellow Torontonian, I'm in!
let's do a asianometry meetup for Toronto EE and IT@@bee1707
The synthesizers we make with these are gonna sound crazy.
Now that is my jam!!!
Elaborate pls @@micro-organism-pv5gd
What synths or effects did you have in mind?
One memristor replace a digital macro component made of:
1) An A/D converter
2) An 8 or 12 bit Flash memory
3) A D/A converter.
The applications of memristor arrays as computing memory really depends on the voltage retention qualities of the device.
Thanks for the video,
Anthony
I really appreciate this. I'd always hear about Memristors online, but not quite get what they actually were. This helps a tonne, and I am thankful for it.
The concept of analog technology in a surprising field does remind me of something - have you ever been interested in what the life and death of the Scanimate machines were? Those analog CGI machines that produced a lot of CGI graphics for TV and film from the late 60s to the mid-80s.
People seem to completely forget they existed when it comes to discussing CGI, and I'd argue they ended up being an essential and important stepping stone for the eventual incredible success of digital CGI.
The way the machines work is both incredibly familiar to how digital CGI worked, but also completely alien. But it's fine if this sort of thing doesn't interest you.
I'm really happy I got to watch this video, it taught me a lot in something I was curious of, but was too afraid to ask. Keep it up, these videos are a highlight for my day, and do genuinely cheer me up.
I remember when they were a big deal around a decade ago.
For my research, they've been used in memory for a while. But they never really worked out as a universal gate.
Now there's been several AI accelerator chips that have been posting insane numbers.
I immediately thought of memristors. It's like they were almost explicitly made for matrix multiplication.
finally, a vid about memristors
A memristor array looks oddly similar to the Apollo 11 memory modules
You had me at " these 'bad-boys' ". My chem profs back at university pointed that out - everything was a 'bad-boy' to me. Glassware, reactants, terms in equations. Bad-boys all. It was used so frequently that a few of them started using it too even. Anyways, sick vid, sick channel. Liked, subd.
During my undergrad my research orientator said that this would be very useful in the future. That was 2018. Here we are five years later and slowly but surely progress is being made. I doubt it will ever reach peak commercial application, mainly due to manufactoring constraints, but sure is a cool niche piece of tech.
Even if it doesn't reach mainstream mass-market use, I can see how the technology may end up being a key-tech in the future: having a semiconductor that can take on more than 2(3) states at the same time, and also include a memory-function within it, may be crucial to develop General Artificial Intelligence.
Your voice sounds much clearer than usual not sure why. Great content as always I have all your notifications on.
Maybe he became AI
I perceive it as calmer, slower, deeper.
@@yr0 either he upgraded his sound setup or that might be it
I noticed that too...
maybe had the mic closer than usual? didn't hear any traffic either
maybe memristors could be used in super speed camera sensors or fusion of neural network and sensor
Current computation based neural networks are just stupidly inefficient, to determine whether an artificial neuron should turn on you have to read a bunch of values from memory, which is not in the same chip so you gotta go through a bunch of memory management crap, wasting a lot of power and time, then you gotta do the math which is basically unnecessary because neurons don't need precise numbers anyway. Then you take your result, put it back out into memory, so it could then be used by another round of simulation of another neuron down the line.
Where as for a real neuron, this whole process is basically just baked into the wire, you send information into the wire, it automatically gets transformed and transferred over to the next neuron, there's no unnecessary nonsense like math or memory.
A big reason why current deep learning based AI is so stupidly inefficient, but if we could somehow have hardware that does the same thing as neurons, without using math or external memory, it would instantly be orders of magnitudes more efficient.
Ok, it's not THAT inefficient. You can divide up layers on different GPUs, so they don't have to fetch weights from off their own VRAM. The most efficient fused kernels, like FlashAttention 2, try to do as much as possible with on-chip cache while minimizing calls to VRAM. But still, a crossbar memristor array would be like 10^6 times more efficient.
Memristors is like Sasquatch, many sightings over the *decades* but none have ever been captured.
Huh? I made a mristor at home. It is the simplest semiconductor to produce. The materials necessary cost $5. What do you mean??
Didn't Huawei just announce they made one like a few weeks or rather months ago?
HP was working on Memristors years ago, but nothing seems to have come from it.
UFOs have been using them since the 1950s
Give it time. Niche applications, if cheap enough. But large computing arrays - er not if it involves too many subtractions of large numbers. Remember the Intel 486 multiply section?
OMG I HAVE BEEN WAITING FOR A VIDEO ABOUT EXACTLY THIS!!!! Ever since Sixty Symbols made their video on memristors 😊
Qinghua University in China built a memristor chip several months ago, and conducted test in AI, and found to be much better than the regular approach. I personally think that by building a Field Trainable Memristor Array (FTMA), similar to the Field Programmable Gate Array, one can achieve AI function very similar to our brain. There is no need to worry about the imprecise nature of the memristor as our brain is also imprecise in nature. Jeff Hawkins of Numemta explained it very well, and has come with highly redundant models similar to our brain that can overcome this shortcoming. Using this approach, AGI can be implemented very quickly, with efficiency and cost very similar to our biological brain. Yes, AGI is finally coming faster than we think possible.
similar to a human brain? Yea, faster and prone to fallibility. Beware what you wish for!
Thank you for show the Doctor Chua's papers on video; I could find it by the name and now I will use it on my work!
This is cool for me because I used to be in a group BEAM memristor group where guys were working on making them at home!
Recently I've learned more about electronics and now there's all this legit research on the topic and it's great stuff.
I used to be part of a BEAM robotics group too, and before it died out, one of the last major discussions was the possible applications of memristors, and there were a few experimenters trying to make them. I think all of us had seen the Nile Steiner video and his home made memristors. I did 2 experimental designs, and did a full concept for a third, that I never tried out. First experiment was an absolute failure. Tried using silver paint to create the electrodes through a pinhole in an insulator over a Copper Sulphide layer over copper. The Copper Sulphide was way too porous for the silver pain and it just soaked in and shorted.
For the second design, I bought 0.6 mm diameter aluminum balls from McMaster Carr to act as point contact electrodes. I created a sandwich between two PC boards, with the bottom layer masked and the copper pads reacted with sulfur to create the Coper Sulphide electrodes. The top layer reused the silver paint as a bonding agent to secure the aluminum balls to the top board. I screwed the boards together, but imperfections meant most of the 16 possible memristors on the board were either not under full contact, or were crushed by the force of the screws joining the boards. I had two of 16 memristors show hysteresis for about 2-3 test cycles before breaking down... a small success, experimentally, but ultimately, another production failure.
My third design, which I wanted to try, but then the first discrete chip based memristors hit the market (though costly). I never followed through, though I've considered going back to it. The design used a thick "sturdy" PC board as a base, and again, silver paint would be used to bond tiny Aluminum balls to Copper pads. these pads would attach to a 20 pin header soldered to the board. a spacer board and a "finger" board would be made from 0.5 mm PC board material. the spacer would support the finger board 0.5 mm above the base PCB, and the fingers would look like a comb with header pins on one end, and thin fingers tipped with a copper pad on the other end. Those pad tips would be reacted with Sulfur to create Copper Sulphide electrodes that would have only the force of flexing an 0.5 mm PCB by 0.1 mm. My hope is the much lighter force would not crush the fragile Copper Sulphide layer, and allow memristance to be observed. By having two rows of 20-pin headers, the part would have the footprint of a 40-pin DIP, be usable on breadboards, and offer 20 memristors in the space of a 1980s era CPU. That was the plan... I didn't learn KiCAD for another 4 years, and since then, KNOWM has released even more memristor options for discrete experimentation. Still spendy, but not as bad as the old Bio Inspired versions from several years ago.
I'm glad this video is covering the more analog capabilities of memristors. So many companies just looked at it as if it were just RAM 2.
I wonder how you learned all that. You make it sound so simple. But if I go out searching for all this I might just end up scratching my head. Amazing videos and amazing work.❤
Thanks for the work, as one of the enlightened I appreciate the effort. Will watch as many times as it takes to soak in in. Feel better, Teacher.
I wonder if the defectivity may be a smaller problem than it looks on the surface.
First, current research into quantized models has shown good-enough performance at inference time all the way down to 4 bits, which would at least seem to suggest a good bit of tolerance for poor precision as long as the overall design of the system takes it into account.
Nice video 🙂, I had this in my mind for research in the final year of my college before HP's patent and research paper came out. Application looked like first principles approach during my college education.
This channel is a pure gem, no cap.
Thanks so much for creating and sharing this informative and timely video. Great job. Keep it up.
The voice quality is much better in this video. I guess the author has finally bough a good microphone:)
Reminds me of that research paper using existing NAND designs but using them in an analog scheme for one of either inference or training.
There were two small start ups working on this in the US in the past few years but one of them went under like a year and a half ago. Definitely promising just needs the right team and right market concept. Edge computing the demand just isn't there yet and it's a lot of infrastructure to build up, I think it'd be smart to start with gaming accelerators - make a PCIe card that has a series of chips for accelerating vocal intonation, NPC attitude, body language, facial expression/animation to line up with speech, all these little things to really make AI use in gaming pop. You probably couldn't load up an LLM, but you could locally turn the flat speech sent by an LLM into dynamic responsive NPC behavior. like a "Narrative Processing Unit" instead of the "GPU" the "NPU" or some crap like that.
It's a concrete goal, you could turn out a decent, usable product and partner with a couple modder communities and big titles to get enough buy in to set up widespread consumer sales, then like a year of work and testing by the team and use the profits from the first run to expand out into the wide variety of areas this technology could be awesome for.
Brings to mind the basic ideas of QLC solid state drives, albeit real time processing with more than a few states is more difficult.
"these bad boys" love it!
Great video bud. But, what about the newsletter? What about the Patreon?
Got thinking about Memristors again after your SRAM video, was thinking about if they could be used in place of SRAM too.
Same, a smart marketing move by Asianometry haha
Hi, not really, memristors (curent Halfnium oxide ones) are very noisy, and have huge failure rates, plus they actually consume a lot of current to be programmed. So they are ideal for ROM, or Inference only accelerators, but for now it is not really though of as cache (PCM or other type of devices are tho).
I thought the same, I think we are a good oxide discovery apart to make that happen.
@@theoballet1519 Of course they aren't good for it now, they're pretty new and underdeveloped compared to SRAM cells. The properties of an ideal memristor would still make them interesting potential alternatives should one ever be developed in a process node compatible with a processor manufacture node.
@@DigitalJedi yeah maybe, we will see with new feram transistors and pcm I guess.
I can't even have words how much i love your videos. The information and the clarity of presenting the information are amazing. Thank you very much thank you a million times
I'm not sure if it works the way I am thinking of it or not. What if you have a material that would change its resistance if you sent a certain voltage or current through it, and it was resetable back to the lower bias state. Then you would have basically flash ram made out of something else. Would it last longer? Could it be as fast a sram?
Leon Chua is a legend in semiconductor circles (and for good reason). However, believe it or not, his daughter is even more famous. That would be Amy Chua. She is the author of “Tiger Mom” (a parent memoir) and the very important (if not well known) book “World On Fire”. She is the author (or co-author) of at least seven books.
Sounds like Tiger Mom didn't quite measure up to her Dad. He invented a world-changing technology that no one really believed could be true. Amy figured out a way to sell books that state the obvious (discipline your kid, democracy isn't always good, some groups make more money than others).
@@joeyp1927 What the father thought of the daughter isn't quite clear. He was opposed to her applying to Harvard (because she would have to leave home) but boasted about her being admitted to Harvard. She has written a number of books of which "Tiger Mom" is just one.
Yes, I referred to her writings on democracy and the differing qualities of groups which are two other books.@@peterschaeffer
To continue the water analogies, I suspect practical, large scale use of memristors is just a pipe dream.
I have actually seen this crossbar device at HP labs in 2018. Did not realize the importance at that time thou.
Intel made an analog NN chip in 1990 based on flash memory cells (aka isolated gate FET). It wasn't a commercial success but I believe it's likely the best approach. Flash cells can be made in any normal CMOS process with very few extra steps. Analog NN really are many orders of magnitude more efficient because you're not shuffling data through your limited number of compute units. Digital NN will also benefit from having compute local to every memory row, but the chip real estate would get enormous.
7:38 9:51 "References and sources go here"
I don't understand how you can create such amazing content week over week!
this is so so great videos, thank you so much! i make my own DIY DSSC now i wonder if i can DIY a simple AI chip with memristor at home with DSSC materials!
I am studying electronics (not on engineer level), and while many of the concepts this channel covers fly way above my head, it is fascinating how relatively short the history of electronics is and how big of an impact it has made on our society.
The way the data is read and written really reminds me of core memory.
Mythic make chips with what sounds like this technology but vs the U55 accelerator its hard to pick a front runner. Great video.
I wonder how much training with dropout (or a similar process modelled on common memristor defects) would help with models running on CMOS chips which may have a few faulty memristors?
Amazing video
Thank you for sharing this information 👏🏻
I attempt to fabricate memresistors at my college clean room for a research project. We used an TiO2 sandwiched between aluminum. The sol-gel was a problem for us
Nyce.
I'm a student who deals in magnetic tunnel junctions which share similarities with memristors. Mind doing a video on them?
memristors have also been tested as SSD and ram but i forget what happened but to say its forgotten not really i think they might use them in IC already as well plus circuit sim has them in there to play with. But use in Analogue AI processing yeah that is new.
I just learn so much from this channel. I definitely understand more about the whole chip business. I’m thinking I could save a struggling chip business….😂😂🍻
In memory analog computations seem more like how organic neurons function
I love memristors and was really sad to see them go the way of the dodo in storage. I also find the possible return of analogue computing in ML fascinating, so to see both combined makes me unreasonably happy 😊
The memristor is a non-linear device with two terminals. You have Diodes and Thyristors that fill this role. Or you can use a transistor and tie the base to the source or drain to form a two terminal device that has non-linear IV curve.
No need for memristor at all.
Somehow superise to see a paper published by our group on a youtube video :p
This was a tough topic and concept to understand at least for me. But you sir described this in great logical chronological detail. That even I could understand it. Basically the flow of electron get sifted through a “movable” variable resistor. Which multiplies it and then gathers those values and adds then giving you an output. Did I get that right I hope. I don’t know why it muliples it maybe I said but I forgot. And why add. What’s the output for. And also how is this memory hwo is it a seeding something that happened in the past to use now. And fyi I thought at the beginning it would work more like the resistance changes depending on the previous resistance needed, and I guess when a new current or voltage would come it change again to that past resistance value leading it to idk change on the fly. Maybe that’s nano tech sci fi but that was just what I though going in. But coming out I now think differently. Again thanks for the breakdown and detailed video thank you and I hope you do more and carry on
This may possibly be useful, but it is definitely extremely interesting! I wonder how long the memory cell will retain its value before it needs to be refreshed.
This is one of those things like oleds where it needs a killer app to receive the research to make it viable to be worth that research. oleds languished for decades as a curiosity until phone lcd backlight display power started being a significant limiting factor in battery life. oleds still haven't even come close to supplanting lcds for larger displays but the chemistry has been done to provide stable organic leds.
I think the concept of a memristor array is going to have a tough fight with in-house AI implementations. The exact nature of the nonlinearity and process variation will put pretty strict requirements on the ADC arrays for resolution, and I think that itself makes for a challenging computational task. If you need a 24 bit ADC to get useful information out of a memristor array when a TPU is using a custom 7 bit float in it's MACs, there's a significant digital tradeoff between a memristor array and "conventional" digital AI training even excluding the analog performance improvements.
Also, is it just me or should there be a DAC on the other side of the memristor input line? DACs are quick and it saves a step in the MAC process.
Also also, This is the same Chua who invented (ish) the Chua circuit, which is cool.
Audio sounds great, better than usual.
Did you upgrade your microphone or apply a background noise filter to clean up the sound signals?
I liked the one before, maybe I am hesitany to change. Or maybe the pop filtering was better before?
Audio is better, even the backgrounds are better
What kind of fab capabilities will be required to produce the first commercial memristors? Will it be limited to TSMC or will some of the Mainland fabs be able to do this kind of research and production?
the Unobtanium fab on the Unobtanium continent has been the prime place to buy them for 20 years.
a lot of money has been poured into them.
I sat in a talk given by HP about 15 years ago on memristors. . A room full of smarty pants’ from everywhere. After an hour I was pretty sure everyone knew less than when they walked in.
Another interesting application of memristors could be storing multiple bits in a single cell, greatly increasing the potential density. This would work by the senising voltage being run through an ADC to determine the bit value of the cell. 4, 8, or even16 bit values in a single cell should be achievable.
A further benefit is that memristors can with further development could also become viable as non-volatile memory with potentially far higher speed than current nand-flash technology.
Multiple bits in a cell? They do it now, the limit is precision. SSD modules deal with that by swapping-in new blocks as they near unreliability/ageing. (a computer on a chip no less) Analogue may be fast, but precision is ALL. Or a moving feast usually. But yes non-volatile, competing with NRAM.
Memristor and nand flash are completely and fundamentally different in basically every aspect.
A nand or nor flash is a so-called charge-trap design that stores electrons, ie. a charge. The amount of charge it can store, and by extent also the number of bits each cell can store is determined mostly by material purity. Even the most optimistic researchers do not expect to nand/nor technology to be able to achieve beyond 8 bit cells.
A memristor doesn't store a charge. It changes resistance. To read it you send a sensing voltage through. This can easily be amplified and run through a flash ADC, limiting only the number of bits by the required speed. Typical 8-12bit flash ADCs can operate in the 10GHz range as used in oscilloscopes.
The other main difference is that memristors can be read/written one cell at a time whereas nand/nor flash can typically only be written in 4-16KiB pages. Making both potential access time and bandwidth speed in a whole different league from nand/nor flash technology.
Super! Thank you very much!
Hi Mr Asianometry, can you please make a video on microelectronics and quantum computing, please?
Sorry to be pedantic, but Von Neumann is to be pronounced the German way, with V sounding as F.
For reference to viewers: at 14:41 BE is Bottom Electrode, TMO is Transition Metal Oxide, TE=Top Electrode and OEL=Oxygen Exchange Layer.
Well,only time will tell.
why not use crystal scintillation for memristors in photonic circuits?
if they can figure out how to combine this with on chip forward-forward training (much simpler than back-prop but still good learning ability), it would be a huge step in using AI in a way that is not so centralized and does not need network connection, while being able to learn as it goes
You reminded me of HP's "The Machine" a memristor based processing system. very exciting until you realise it was just a marketing tool to get more money coming in via shares. It quietly disappeared a couple of years after the hoopla surrounding its release.
I wonder why nobody mentions the first application of memristors in 1939 by Hewlett in his Version of a Wien Bridge Generator. This dynamic memory- resistor was in use until the 1980s and is known as historical circuit.
One consequence of analog AI is emergent individuality.
I.e environmental & process dependent personality or noise amplified random behaviors.
Oh wow. This brings back memories. Please pardon the pun. lol But, no, I remember this debate. I remember being surprised that it got so much traction. It was an epic nerd fight, but I remember at the time concluding that the fight wasn't about anything at all important or interesting. To me, anyway.
Thanks. A lot of your videos are great nostalgia hits. I didn't get to Asia (Tokyo, mostly) until the early aughts. But I was in Europe a lot before that, and worked for and with a lot of companies that came out of Mountain View and Palo Alto and other places. Software.
As I age further into the back-half of my century, it's fascinating to see history's consideration of my past environment. An experience that latter-middle-aged people surely experience far more intensely today than their forebears, because everything's moving so fast. Then again, everything's relative, I suppose. These brains are curiously flexible that way.
the nature of consciousness will never be resolved but when we can replicate a brains function in wires we can arise many questions and make some of the oldest simpler and some of the oldest worse
Hey partner you okay. Because you sound a little off. It is appreciated that you post the video on schedule like normal. But if you need to take a break. I think we can all agree that that would be okay..
This was recorded 2 months ago ^^
Wow
This is HUGE! 🤓
So, in the near future we'll literally have pre trained AI models running on our devices not just on one giant power hungry, super-computer that the whole world shares!
(with possibility to change the weights later on, for example to update your AI assistant)
as long as we're looking at computing in memory for AI, what about memristor-adjacent ideas like Mythic AI using flash memory cells as the variable resistors in a similar array.
Hey, so... If you could get a memristor to quickly react to a high or low current and match it, wouldn't that just make... Ram SSDs? Like, actually. Could also be really good for CPU/GPU cache.
Edit #1 : I know memristor are, by nature, already RAM - but _analog_ RAM. Doing what I suggested could make it binary, thus much more suited for non volatile RAM and cache. Like, could you imagine having a power outage, and when your PC turns back on everything's exactly as it were before it crashed?
Edit #3 : Actually, why limit it to "0" And "1"? By reading multiple levels (Let's say 0 to 3) you could have incredibly dense memory/cache (in this example case, 2*2*2*2 so 16 bits of data) similarly to how high density SSDs already do with different charge levels. Honestly, the memristor could probably fix the "cache apocalypse" of TSMC's 3N and below nodes since they're a single transistor instead of many, even if they might be a little slower at first.
It sounded to me like an electric spark can change the resistance and each memristor value needs to be set at each power on for a NN application.
memristors retain their value when the power is off. In fact you set the value and then turn off the current in normal operation.
Super exciting tech! I believe analog hardware will be ideal for AI because high precision isn't necessary.
1:00 What else Mosfet of Flash memory is if not memristor! Non linear, we might use linear part o characteristic all depend on charge
What i want to say - memristor is magnetic, flash uses electric field requires voltage pump migh be used to hold analog data Holtek once manufactured analog flash memory to record voice messages
We're still expecting some breakthroughs in materials science.😎
7 bit flash cells have been prototyped - that's 128 levels - more than enough for limited precision ai netowrks - could a custom flash cell biasing network allow for a similar AI circuit?
If memristors depend so much in the fabrication yield, then why don't they implement a FinFET approach, which already looks similar to the memristors presented in the video, in order to effectively increase the probability that a single fin is correctly manufactured? Does this make sense?
Is this what Veritasium's video on analog computer were??
Neural network of human brain is not analog. It is basically digital: A neuron can be activated or deactivated like a digital switch, not in-between. But it has variable number of input signal threshold to activate the neuron. kind of summing OP-amp/comparator. Resulting circuit work like analog circuit. Digital CMOS circuit can behave similarly. A array of switch coupled with digital memory cell also can work like summing OP amp. Not computing in digital binary number, but summing number of active inputs in analog way, though inefficient and low precision.
serious stuff
7:25 yes
I discussed this with a top AI scientist back in grad school almost twenty years ago…
The concept is wonderful, in concept. Reality has to be factored in. And the biggest factor is what limits all analogue computing. Precision. Particularly devices that have non-linear properties. The Intel 486 multiply section had a small digital error. Only detectable when you start to compound arithmetic cycles, like subtracting large numbers to get relevant small differences. Accuracy is then ALL. Analogue may be faster - "but" once it gets complex ! Think human mental arithmetic. Large numbers. Any operation. How good are you?
@@giusdbg the techniques that allow digital to run fast will allow memristors to run fast enough to compete with an algorithm favourably. Horses for courses. Precision and complexity will be the digital domain. And multiplying with a memristor is basically "parameter 1 set R, then pass current for parameter 2". easy IMNSHO, but slower. The matrix addressing is by definition digital. So in reality hybrid. As for logarithms, diode have been made with such characteristics.
Are you using a trained TTS model? I notice some weird enunciation compared to your usual narration. Great vid though :)
Wasn't darpa building these building 3d memristor chips?
Why need AI if have project Cybersyn new type economy run by organic I meant human .type economy base from computer
I hate to break it to you but it hasn’t just been over 30 years since 1971, it’s been over 50
What happened to the HP memristor? Why isn’t this a thing? Seems odd that such a historical behemoth would have just left this on the table.
Can anyone please explain how the boundaries drift by applying current
Stacking these would be insane, as memristors are incredibly power efficient. You could have something orders of magnitude more powerful than the brain in a deck of cards…
HP did it in The Machine. Idk why it didn't go nowhere.
Are defects really that problematic, because values in neural network applications are kind of redundant, a few changed values here and there won't change outcome much.
Resistor USE energy if got too much current
I'm intrigued but I fail to fully understand the concept and the implications.
Neural networks do a lot of math, particularly a lot of matrix * vector operations.
In a regular, digital computer, this would be done sequentially, multiplication after multiplication, which can take "long". Even on parallel hardware this could take multiple cycles (I think).
Memristor arrays could simplify the process by just having the input vector as a set of voltages on the top lines and reading the result of the matrix multiplication in the bottom lines. All memristors are working in parallel, making the process almost instantaneous.
@@zokalyx - TY. Is it the same idea as that of physical (hardware) neural networks that I've seen somehwere that is a (potential, under research) alternative to virtual (software) ones, which are the ones being used now in AIs?
@@LuisAldamiz Yeah, in some papers, they named it 'Hardware' NN to differentiate against regular 'software' ANN running on regular digital ALUs. The physical properties of the material (in this case, electrical conductance) and the linear law (ohm's law and Kirchoff's current law) is used here. For example, to do 4x2, 0.4V multiply by 2 microSiemens = 0.8microAmps. 0.8microAmps as an analog value, can be converted into digital 8. 4x2 = 8! Kirchoff's current law is just to add up more matrix elements
@@timng9104 TY for the reply. However I recall that one characteristic of these brain-like (under research) physical neural networks was also to imitate the way brains work, which is "slow", "low energy" and "chaotic" (as opposed to the heavily centralized CPU architectures). This is, I understand, not necessarily what a "simple" shift from transistor (CPU & RAM) to memristor architecture but retaining the current mainline "neural network" paradigm would do, right?
IDK, probably there are several parallel and somewhat related lines of research here, all cutting-edge and thus a bit hard to understand. If they succeed, they may be behind a new AI and overall computing revolution (for the good and for the bad). Or it may not work at all, like the alleged graphene revolution and such.
like a new kind of a fpga.
it's not the only use case if someone managed to produce them though.
I thought my research topic wasnt that popular,i guess i was wrong
Coils, yay! Next comes radio. ;*[} (or rather next comes analog ML...) And phase change sounds slow....like burning DVDs... we'll see - cheers.
I heard they don't work because they are not stable at varying temperatures. Dunno if this is true or not.
Non-linear two-pole devices already exist, in the form of diodes (change when you overcome the forward voltage) and ... fuses (change permanently if you pass a too high current through it). I'd think the fuse better reflects the sand analogy and the memristor sandwich looks remarkably like a diode.