Storage (SD Card Support) - IO from Scratch - Part 8
Vložit
- čas přidán 25. 06. 2024
- I’ve been wanting to add some kind of persistent storage to the build. In this video I discuss various options for persistent storage before building the interface to interface a 3.3v SD card to the system.
Most of the work to interface to the SD card is in the coding, I have an abridged version of the coding in this video, and a longer edit with more detail on the extras channel video here: • SD Card Coding - IO fr...
0:00 Introduction
0:35 Selection Criteria
1:46 Device Options
2:48 Selection Grid
3:42 SD Cards
4:21 Level Shifting to 3.3v
6:22 3.3v Power!
7:44 Logic Level Shifting
11:07 Wiring the SD Card
12:19 Code Introduction
13:55 A Choice
14:17 SD_Init()
19:30 SD_ReadBlock()
21:52 SD_WriteBlock()
23:53 Final Test
25:00 Outro
George Foot’s floppy drive videos: • Homebrew Floppy Disk I...
The Digikey Level shifting article: www.digikey.co.uk/en/blog/log...
The Tutorial I was loosely following: www.rjhcoding.com/avrc-sd-inte... - Věda a technologie
![Eminem - Houdini [Official Music Video]](http://i.ytimg.com/vi/22tVWwmTie8/mqdefault.jpg)
Join us on Discord: discord.gg/jmf6M3z7XS
Follow me on Twitter: twitter.com/WeirdBoyJim
Support the channel on Patreon: www.patreon.com/JamesSharman
The magic number of 1563 for "Max read attempts" was indeed a specific timeout. The spec gives 100ms timeout. My loop for checking that is 33 cycles so at 4mhz my value is 400000 / 33 = 12,121.
Really like your new editing style where you show building the schematic and the breadboard at the same time!
I too think that's really cool.. drives my multi tasking capabilities to it's limits though 😄
I've done that on a number of videos where I feel it's needed for the viewer to keep track of what I'm doing.
eagerly waiting for file system part, thanks:)
Indeed File system support will let me add a lot of functionality.
What I’m thinking if I build something like this using Forth would be to just do raw block numbers for reading and writing which is how Forth does it normally. But you can build file system support on top of that since Forth lets you build up abstractions quickly
Buffer chips can be used for level shifting. 74LVC family are 3,3V chips with 5V tolerant inputs. 74HCT can take 3,3V logic and outputs 5V. So for example, 74LVC125 can be used for conversion 5V->3,3V and 74HCT125 for conversion 3,3V->5V.
Also make sure to use low-dropout voltage regulatr. The most popular choice seems to be LM1117-3.3
I think the HT7333 is better. It has a very low drop. He just doesn't like short circuits.
The LM1117-3.3 is cheaper but not better. It just has a low drop voltage worse than the HT73xx.
@@jensschroder8214 Surely there are better voltage regulators. LM1117-3.3 is alright for regulating 5V to 3.3V.
I don't know the 7833 regulator he used, but if it's based on the oldschool 78xx family, these require at least 2V voltage drop for stable operation.
I did mention there are lots of options, I was trying to go simple and explainable here.
@@JendaLinda You got it backwards, all the various 7333 models I found all have MUCH lower drop-out voltages than any LM1117 model I can find, you could call them an ultra-low drop-out regulators if you want. Depending on manufacturer it's usually in the 100-200mV range, compared to the 1.0-1.2V the LM1117-3.3 need depending on manufacturer. Lets just say that a lot has happened since the LM1117 was introduced back in 1998? (or earlier, but it was definitely available by then).
@@weirdboyjim That's fair. I just thoght using various types of 74-series chips would be still in the spirit of simplicity and perhaps an opportunity to explore different logic families.
+1 for George Foot. Does a really great job on covering the subject
He's a smart guy.
Great video. Just what I needed to chill after a long day. The schematic and breadboard at the same time was helpful
Glad you like it! I do try and do that when I think it's needed to clarifying what I'm doing.
There needs to be a second SD card socket so it can operate as a dual drive system. This will fix those pesky disk swaps when copying files from one disk to another.
Would actually be quite easy to add that as an additional SDI device, but it would need to be with a fly cable, no room on the pcb for another socket.
@@weirdboyjimadd a daughter board for the second socket and hover one above the other
It is somehow incomplete without a floppy. I remember very well two episodes of my early computer life: 1. I purchased a 20 MB hard drive, and my mate asked me what I am going to do with all that storage space. 2. I had audio tapes for storage with my eastern Germany KC85, and got floppy later on. This was so huge, in terms of storage AND speed of operation.
Nice job with the SPI and SD
Thanks Peter! In my early coder years the transition from audio tape to floppy was magical!
Thanks for the mention - I think you were very generous not putting a big "X" for floppy drive parts (and media!) availability, as they are scarce, expensive, and fragile!
For me SD cards are a great accessible solution, but there came a point when having to plug and unplug the card was getting frustrating, and I've mostly used a fast serial interface since then, tethered to a PC. That said, I think there's also a happy medium where you use serial to send data which your homebrew system writes to the SD card, which could give nice fast read speeds from SD card without having to unplug/replug/unmount etc the SD filesystem.
You are welcome George, your stuff is excellent!
I do a lot of code development with the simulator, but that doesn't really help when implementing new hardware features. It's all about getting the iteration time down. Alongside the binary my assembler spits out a ".up" file that contains the commands necessary to upload and execute the code, so I just copy paste that into the terminal window.
Right I see, rather than pasting, my last couple of systems have used a special protocol to tell the PC to send a named file, which allows for multistage booting, and loading arbitrary apps. So on the host side the build process just puts all the binaries in a directory and the client code can access them from there. All very unfinished though!
Yay! Two of my favourite CZcamsrs! 👌
This is gonna elevate the project to a whole new level, since the console/bootloader could hotload programs from a raw file AND have persistence. Also, it's a convenient way of storing your demos, no more reboot/copy/paste
I'm looking forwards to adding lots of functionality. If I can get the monitor loading programs from SD I can get all my old test code and utilities onto an SD for easy access!
I like how you don't make the fast-forward part silent but just leave this electronic hum. Makes such an atmosphere...
I fill the gap with a section of "silence" near by, it's far less jarring than true silence. Would make less of a difference if I could clean up the base audio more.
I've been working with SD cards in embedded systems recently. Protocol support, timing compliance, and general quality is *really* all over the shop. There are tons of fakes out there that only support a tiny range of the various interface modes.
When you have problems with SD cards: try changing: card suppliers, protocol settings, capacitors; in roughly that order.
Good luck!
I've heard that. I want to test everything I do on a wide variety of cards, my current code works on all the V1 cards I've tried it on.
Level shifting is one reason I have so many unfinished projects. I'm loath to add another chip with all the extra wiring so I let myself get distracted with other things instead.
For very simple things with one or two separate lines I've used a resistor divider for step down and a 2n2222 transistor for step up which works well enough.
One one project, I've been 'cheating' by lowing the 5v supply voltage enough to bring the 3.3v high logic level up enough to work as a high level for the 5v parts of the project, but it's a really dirty hack.
Anyway, great stuff James, really great to see both the logic level shifting and the low level SD card access. Looking forward to the evolution of this system.
The more I think about a resistive divider the more I think you really should have a buffer before the device. No idea what you are driving!
@@weirdboyjim None of my stuff is commercial quality and when I let the magic smoke out it's my own fault :)
This is amazing. Congratulations James 🎉
Thanks Leo, glad you liked it!
Thank you for another amazing video, James!
Glad you enjoyed it!
Impressive! The SDCard prootcol always felt a bit opaque to me. Now the veils have lifted a bit ;-).
I really need to get around to a storage solution for my own project as well, but right now my creative output (if you can call it that, ha ha) outpaces my capacity to have it built and assembled. Maybe it's time for me to step back a little and look at what I've got ;-).
Nice! I've been wanting to do exactly this for a while now. I'll definitely be using you as a reference if I ever get to that project.
I hope it's useful when you get around to it!
You're going to want a write through cache very soon! Maybe even two sdcard slots so you can interleave writes between two very slow interfaces to pretend to have one medium-slow interface
Omg this is amazing 🤩🤩🤩
Glad you like it!
This will be interesting! My little FPGA homebrew has microSD for storage for the simple reason that the funky lil SPI TFT display it has came with a bonus microSD card slot on the same SPI bus, so it's a given I would take advantage. But it's not something I sought out - seeing actual logic and reasoning that brings you to using SD with all the options you have is going to be really interesting!
and it turned out dead simple. Perfect!
Glad you liked it!
Have you considered writing a C compiler for your system? As you're dealing with more complex stuff, assembly is going to become painful. There must exist some tiny C compiler which you could re-use and write a custom backend for your CPU.
NICE!
Yeah... my first thought would be USB, then it could even have stonking great hard drives... But trying to fit a USB software stack on an 8-bit machine is just about possible but kinda massive... and building the interface from discrete logic would be... ... ... "interesting". SPI and SD cards are a clear winner there! (George Foot's videos are great, you've reminded me to go back there and see what he's up to)
Maybe on a future build with more memory....
Thanks! I'm learning FPGA. I'm going to start work on the SD card after I understand HDMI. This will help.
Interesting, HDMI requires a far higher bitrate than I;m dealing with here.
@@weirdboyjim I'm learning how the sample program works. I can move a square across the screen. I setup a 3,000,000 pixel clock delay to slow it down.
I've had to deal with a similar voltage level problem (connecting my ben eater style clock to a 3.3v FPGA). You can improve a tiny bit on the digikey solution which I also used as inspiration. That design assumes a bidirectional tri-state bus. But if you have a line that is constantly asserted by one of the sides (let's say your SCLK, oscillating between 5V and 0V), that 10K going to vcc will not change anything. You can get rid of it and save yourself a tiny bit of PCB real estate and routing.
Thanks for your video series by the way! I've binging on them for some time and I'm really happy with this one which is the first one i can see as it comes out
Glad you are enjoying the videos! This video was not "about" the 3.3v connection so I wanted a solution that would be easy to explain and then move on. The 5v->3v lines can have several shortcuts but that does start to assume what is generating the signal so I would either need to leave things unsaid or go on a bit of a tangent.
I've used the same circuit as the one shown from Digikey (mostly for i²c) for years, using 2N7000s and it works very well indeed!
Ah the good old putting it back to front, the number of times I forgot I did that... lol
People have suggested I shouldn't use the 2N7000, I've ordered some of the others, I'll report back when I can.
@@weirdboyjim yeah they're not logic level iirc, it worked fine for me since the current is so low (and I have a metric sh..load of them), but I have to agree with People on that one :)
Btw. it is possible to just solder the needed 4 wires to a SD to microSD card adapter (and use a microSD card).. in case one is really stingy with the moneros.. or needs it "right now" and has those parts.. worked fine on one of my esp projects 😅
Fascinating video as always, thanks a lot !
That would indeed work, that little adapter is once I made myself, let's me be sure it will import into the final pcb correctly.
Really useful thanks! I'm currently using an odd storage option and that is Microchip serial EEPROMS. 32k/64k chips all using SPI and at 5v which makes for easy hook up. I can then externally use my EEPROM programmer to read and write too.
I did consider an SPI Eeprom, maybe as a hard drive with the sd acting more like floppy.
@@weirdboyjim Worth a look at. With mine I'm using them as a number of different 32/64k "drives" with a simple file system on top. The SPI io is really simple with a 64byte block size. So I have five of these onboard (the other three pins for SPI bus) on one side of my Z80 PIO. The other 8bit port of the PIO Im using as select lines for a removable storage of these. Capacity then is around the size of an old 5 1/4" floppy. Nice to try something different :-)
A bit more robust (and slightly faster, though you don't need that, most likely) solution is the SN74HC245N. Comes in DIP format as well as surface-mount versions.
To do it with those, you would really want 2 of them 1 running at each of the power levels. The one converting from 3.3 to 5 would want to be a HCT part so the data line from the 3.3v device was sure to be other threshold.
Compact Flash is also an modern option
Ticks many of the same boxes as SD cards, but it's physically larger interface which put me of (I'm pushed for space)
Hang on, was the RTC work just so you could get an accurate clock for the SD card reader functionality? Or did the stars just line up? Either way, very well done. I am off to watch the extras video, where the real party is.
Nah, I could have got that delay by counting cycles but it was nice to use the RTC. I'll be needing that for file system stuff though,
Nice, I like that you do it from scratch instead of just adding a microcontroller to do all the stuff for you. Another option would have been a Compact-Flash card as a simple PATA Device mapped to some port addresses for PIO. The bigger "fun" will be to implement something like FAT16 on top of the hardware.
Starting to suspect I should have included CF as an option. CF does tick many of the same boxes as spi but it's physical larger than SD and I didn't have a good place to put it!
@@weirdboyjim i second the PATA option - no level translation, only 2-3 parts for the interface, period appropriate, and most importantly: you can plug in actual spinning rust!
This is huge !
One step closer to a portable game system perhaps ? ;)
Thanks! Game system maybe, not sure about the portable ;-)
IDE (“PATA”) and an optional Compact Flash adapter would have been viable, though they are neither 8 bit appropriate nor current (AFAIK). If you couldn’t find cards old enough to support the 8 bit mode you’d need some latches for the high half of the 16 bit bus. On the plus side IDE is fairly trivial. But being selfish I’ve done IDE on 8 and 16/32 but home builds but not SD, so this video is more useful to me. 😂😂😂
Be careful with 2N7000's. There's a reason the diagram your showed are using BS170.
The Vgson(th) is considerably different for both FETs, and 2N7000 aren't reliable with 3.3V logic, let alone 1.7V (5.0V-3.3V).
You're likely just using the body diode.
Interesting. The circuit is fairly common online, the first I found used the 2N7000's but I went with the DigiKey article because the description was better.
@@weirdboyjim oh they're super common, with both FETs. But differing amounts of acceptability. That circuit is also prone to oscillating in certain conditions. Usually at a few MHz. Just keep it in mind 🙂
Great video as always, James! Keen to see how the filesystem works out!
Why not use CF-Cards most of them can be used in 8bit mode and they have parallel interface, interfacing them should be really easy with few registers and busdrivers. Ciernioo have writen nice article about interfacing with Z80. And on pjrc is extensive writeup on how IDE parallel interface works and how to interface it with 8051.
also interfacing with modern pc should not be problem
Is quite an uotdated technology though to be honest? Micodrives would be a fun quirky option on that theme though.
@@rimmersbryggeri CF cards are still made and used in devices!
+1 on CF, curious on how that would've stacked up in that chart
@@colinstu I havent seen a new device that used CF in a long time though. Last were dslr cameras that seemed to use them for a long time. I suppose CF is good though since it is somewhat SATA compatible if I remember rightly.
@@rimmersbryggeri CF is basically just miniaturized IDE. So, IDE compatible.
Many _many_ years ago I actually wrote a FAT12 image reader/writer in QBasic.
It wasn't very hard.
And before you ask - no, I don't have the code anymore.
Argh, Fat12 would be a pain. Fast16 is the sweet spot for implementation on an 8-bit cpu. I don't expect it to be particularly challenging but it's interesting enough to make a video out of.
@@weirdboyjim from what I remember, not much about fat12 is actually 12 bit (actually, I'm not sure anything is?) it might o ly get you to 4MB though.
It was used on floppy disks
Only slightly more elegant than plugging in an eeprom but I think cartridges deserve a spot on that list. Wouldn't it be funny if it escalated to an SD-based flash-cart?
It would feel like a game system, but I'd need to build something separate to plug them into a pc.
My suggestion for filesystems is to keep at FAT16 or FAT32. I think older DOS versions had FAT8 (which would be perfect here) but I'm not sure if any modern OS supports it nowadays. Other than dealing with 16 or 32 bits, FAT with only 8.3 filenames are super easy to parse. BUT you won't be use all 128GB of your SD. FAT16 is limited to 2~4GB and FAT32 goes up to 32GB. Also, if you want to use smaller cluster sizes (ex: 512 byte blocks), then limits go even lower.
The older dos one was Fat12 which is a little annoying due to cluster indices not being a whole number of bytes. Fat16 is probably the sweet spot for an 8-bit system. With a 4k cluster that gets you to just under 256meg.
It seems to be getting closer to the time that a DMA device for RAM->RAM, SPI->RAM and RAM->SPI would be really beneficial now. Being able to stream data into RAM and vice versa would allow audio streaming, or background loading of assets etc :) Would also be handy for the video side with sprite blitting! I can't remember if you have an interrupt available though.
For DMA you'll need to wait for a future build although I suspect you would be surprised how many system calls even on fairly modern platforms have low level oops like this.
@@weirdboyjim Yeah, it varies. I do PCB design and FPGA design work, and made a DMA peripheral for the ZX spectrum expansion slot a couple of years back, to allow far more sprites and scrolling backgrounds driven by the DMA, using a microcontroller to populate the RAM on the cart. Didn't seem much point continuing it though, as it kind of gets away from the reason people program for the speccy - to try to squeeze the maximum performance from the hardware. I almost worked with the guy who designed the Spectrum Next (for his current project), he's working with my friend (who wrote the J2ME port of Jetset Willy, and Jetset racing) over in Switzerland - or at least was, last I checked :)
I've been making a little game engine with composite out for the old 8-bit Arduino UNO, and ported a couple of arcade games and Manic Miner across to it recently for fun. It has scrolling tilemaps, masked sprites at 256x256 currently, with just a couple of resistors for the output - made a little shield to add an NES controller port, and make it easier to connect to a monitor, but it's all done by racing the beam, using the UART for video output :) I find it therapeutic after spending most of my time with chips that are way overpowered for what they're used for...
For the 8-bit era similar to that of the floppy drive but different; what would your take end up being towards using the zip drives that were common during the late 80s and early 90s? Or were they more towards the 16-32-bit era of computing? Also, if you chose to work with something like that how would you go about in designing an interface to work with such a device? I think that would be kind of interesting to see. It's not just the fact of adding storage to your system, but also the ability to compress and uncompress your data to and from an external I/O medium.
I had a zip drive back in the day, but the drives and media are not easy to get these days.
@@weirdboyjim True, yet it would make for an interesting project, just from a retro-historical perspective alone. And yet due to their limited availability is why I completely understand that you went with the SD Card option.
In the video I see using the open drain logic level translator which is common for IIC. As far I know SPI is push/pull level. As well some out-of-box IC level translators are suitable or not for SPI or I2C. Question: can we really use the open-drain level translator for SPI too?
And - nice low level programming for the SD card :)
Interesting questions, but it's generally best not to assume. The data line requires a pull up, which makes me feel it maybe open collector in which case a pull up and diode would do it.
I use SD breakout boards with built in level shifter chip.
I couldn't find one with a real left shifter with a full size SD card socket. I'd want to do it myself for the main circuit anyway.
Nice, will the filesysterm be part of an OS? I don't think you should call it JAM-OS, Marmalade has more appeal... I'll let myself out. Thanks James. Take care. 🍊
Thanks Jerril! I've been thinking about what use to make of the filesystem. Adding support for executable loading to the monitor would indeed make it closer to an OS.
@@weirdboyjim Tiny Operations And System Tools: TOAST!
Floppy is still a good medium, long-lasting and very transferrable.
So from your test, you already formatted the card and created the file then just tweaked known blocks used in that file?
For that test yes. I've implemented init, block read and block write. I'll use those in a future video to access a file system.
@@weirdboyjim yeah that is a toughie - can't wait for that vid. Cheers.
No tutorials for implementing an SD driver for the Jam-1? These tutorial writers are sleeping on the job! 😆
I know right!
I hope you make it a cartridge like the EverDrive carts for Megadrive and SNES that you plug an sd card into.
I remember there was something like that as a replacement for the Sinclair microdrive!
No Hawk or Phoenix drive support? Or at least memory core. Just kidding, SD is probably the best choice today. Raw FS is probably usable but I'm guessing FAT12 is next anyway.
Every time I tried to get one they said @UsagiElectric had already grabbed it! Lol, filesystem support will be soon but I'll be working with fat16 rather than fat12 I need the extra indices for larger cards.
would make for an interesting virus scanning system that has a very special processor impervious to your average malware
Lol, indeed. I can honestly say I've not seen Jam-1 malware in the wild yet!
@@weirdboyjim is that a challenge ? 🤣
Beware 7002 mosfet is not guaranteed to open . Better use bs170 with guaranteed low voltage threshold.
Someone else suggested that, I've ordered some and I'll give them a try.
Was wondering when the required FAT was going to rear itself. And wear levelling etc.
Filesystem access will make for some interesting new possibilities!
@@weirdboyjim Well in the old days it was just sort of a list of pointers to linked lists.
the table of contents pointed to the first block and each block pointed to the next. until you got to the end of the file. Somewhat over simplified description. maybe a reverse engineering of FAT16 is called for. 🙂
How much faster this computer can read from SD card than from UART?
UART is roughly 10KB per second. SD transfer rate is 0.5MB
I wouldn't regard floppy disks as a viable option for anything other than supporting old gear and definitely not for new, as nobody makes them anymore. Fine if you've been into this for years and you have a stock otherwise a great barrier to those new to it.
Audio tape.. interesting as there is software out there to talk this from PC's as well as mobile devices, however you did complain about slow serial, well audio takes slow to a whole new level.
Punched tape, a reader is not that hard, however a punch... wow.
USB, well that and WiFi have to be the most convoluted systems ever, with many using ESP and similar micro controllers to bridge the gap - now that I find very inappropriate as often these chips can easily emulate the whole 8 bit system your building, kind of making it all pointless if your going to put that much grunt into a project just for I/O.
As for the FET level shifter, everyone seems to love these, however they are kind of like open collector type drivers, not bad for slow speed, but wind it up and slow rise times start to come in, that's why logic went to tottempole style active pull up and down years ago to get the speed, so this is kind of a crappy solution.