Email vs Capitalism, or, Why We Can't Have Nice Things - Dylan Beattie - NDC Oslo 2023
Vložit
- čas přidán 24. 06. 2023
- We're not quite sure exactly when email was invented. Sometime around 1971. We do know exactly when spam was invented: May 3rd, 1978, when Gary Thuerk emailed 400 people an advertisement for DEC computers. It made a lot of people very angry... but it also sold a few computers, and so junk email was born.
Fast forward half a century, and the relationship between email and commerce has never been more complicated. In one sense, the utopian ideal of free, decentralised, electronic communication has come true. Email is the ultimate cross-network, cross-platform communication protocol. In another sense, it's an arms race: mail providers and ISPs implement ever more stringent checks and policies to prevent junk mail, and if that means the occasional important message gets sent to junk by mistake, then hey, no big deal - until you're sending out event tickets and discover that every company who uses Mimecast has decided your mail relay is sending junk. Marketing teams want beautiful, colourful, responsive emails, but their customers' mail clients are still using a subset of HTML 3.2 that doesn't even support CSS rules. And let's not even get started on how you design an email when half your readers will be using "dark mode" so everything ends up on a black background.
Email is too big to change, too broken to fix... and too important to ignore. So let's look at what we need to know to get it right. We'll learn about DNS, about MX and DKIM and SPF records. We'll learn about how MIME actually works (and what happens when it doesn't). We'll learn about tools like Papercut, Mailtrap, Mailjet, Foundation, and how to incorporate them into your development process. If you're lucky, you'll even learn about UTF-7, the most cursed encoding in the history of information systems. Modern email is hacks top of hacks on top of hacks... but, hey, it's also how you got your ticket to be here today, so why not come along and find out how it actually works?
Check out our new channel:
NDC Clips:
@ndcclips
Check out more of our featured speakers and talks at
ndcconferences.com/
ndcoslo.com/ - Věda a technologie
Folks, the thing missing from the end of the talk is one person's email address, which I thought was funny, and wanted to share with the audience, but didn't want to put on CZcams. It's about 45 seconds of content. That's it. No massive conspiracy. No secret discussion of Google, Outlook, etc... it's was just a fun little joke to end on that I didn't want to end up on CZcams.
No, I'm not going to tell you what it is. But it involves the top-level domain for Austria, in case you have a hunch what it might be. ;)
Oh, and those of you saying "but where's the bit about capitalism?" It's the bit of the talk where I talk about unsolicited commercial emails, spam filtering, the influence of marketing on digital communication... y'know?
No massive conspiracy? Ahhhh.... Where are we going to get enough anger to find motivation to stop turning a blind eye to everything?
one the slide with the zoho email where they say, relay not allowed, you accidently still had the protonmail host part in the rcpt to.
I think the + feature is fairly common, I know I've used it on Microsoft 365 at least
Thank you for that. Really wanted to like but you already had 69 of them so couldn't
I am so not going to tell anyone else about this video because doing that would enable them to understand email so easily and I would be jealous of them! Just kidding. Totally gonaa share it with my peers. This is the most astoundingly simple explanation of email....kinda like the one you had with "Plain Text" which I absolutely loved!
I wish we could go back to when the Air Force threatened people for sending spam.
Use the navy - still a large air force and they can float airfields around the world...
Yeah dude just tell the Navy to do it 😂
If only somebody back then would have invented reusable proof of work we could have fixed the problem from the beginning.
@@irgendwieanders2121 Send in the Marines!
I don't consider it spam if the from address is for real and not changed with each email sent.
I ran an email security / antispam company for 19 years. We saw the most horrible abuses of standards you could imagine, and had lots of special-case code to handle them. (I'm now happily retired...)
Haha, as I watch this video, I recall the endless arguments I had with people who "validated" email addresses with code that rejected valid email addresses.
If there's one thing you can guarantee about standards, it's that no matter how clearly and precisely you define anything, you will still get a raft of different implementations from companies who all insist that they follow the standard.
I started programming last year.
A Mail validity checker would seem to me like a bunch of if/elif/and/or statements (like a logical statement) and to me it seems not that difficult.
But when I’ve learned something in programming then things are hole lot more complicated! What is it with email validity checker?
@@vibue2415 it needs to be recursive and it's VERY complex. Most people who try to write them have also never read the standard. Probably don't even know it exists.
@@photinodecay i've seen an email validator that didnt work cause the email was on a subdomain
"Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."
Terry Pratchett, Thief of Time
Yes! That's it. I thought it might have been Douglas Adams but couldn't remember for certain where it was from...
GNU Terry Pratchett
@@wrongtown GNU Terry Pratchett
@@electroflame6188GNU Terry Pratchett
When you mentioned the problem with dark mode, all I could think was “well, let’s stop with the annoying formatted ‘pretty’ emails, and have them just send text”. I’m already annoyed with logos etc in emails I receive
About dark-themed email clients: the most important thing, designers must remember: its unacceptable to define color just for fonts or background. If you are altering colors, you MUST specify BOTH of them: background AND foreground. Both or neither! Because you dont know whick colors are default in viewer.
Back in the day, I had a richard for a team lead. I wanted to go to every site I had control of and give him fore and back colors as the same.
You also don't know which one the customer's software is configured to override. Could be both, even. So even if you specify both it could look totally different.
Better yet: Dont specify either in email. Respect your clients.
Amen, @@Carewolf.
I'm a dinosaur, and I had all my system colors set to a custom dark theme back in the Windows98 to XP days, before it became a thing. Easily 75% of web form input fields had the color or background explicitly set to the default black and white, respectively. This made everything white on white or black on black. This was so irritating, and I dealt with it for years. Thankfully things improved.
Enlightening and entertaining as always!
Minor nitpick: When you telnet'ed into your Zoho SMTP server, you supplied a Proton Mail recipient, and rightfully got a "relaying disallowed" message (28:45)
I was just about to comment this!
From what I can tell this domain is installed at all these providers. You can do that and it's not an issue but whatever DNS responds with will be your "primary" provider. This is why the telnet to the providers, it forces the initial connection open to each provider.
@@Iamdebug The Zoho SMTP server would not (and should not) accept incoming emails to a Proton recipient.
Otherwise, it would be an open relay. A somewhat common misconfiguration back when, others than service providers, ran mail servers.
That's not at all what I said. Nothing in the footage I saw showed Proton at Zoho. All domains are installed in all providers and providers are isolated, they simply don't care if another provider has your domains also installed. If DNS doesn't forward you to it the other systems essentially doesn't exist. @@Woolbox
I love how you demistify email. From a techie point of view I always heard more senior people telling me how hard it was to understand email and smtp and dkim and so on, and I went to believe that. You just broke that apart plus added an history lesson above.
Thanks a lot!
Dylan is great, but don‘t fall for the Dunning Krueger Effect. This talk scraped one atom layer of a one ton block of email mess 😮💨
@@dinoscheidt , definitely. In-depth email configuration of the many hacks, bugs, dealing with misconfiguration of other servers, and so on, could fill hundreds of hours of video.
He didn't break it apart. He tied together a thin layer of narrative that fits his presentation.
MKBHD's video of the secret history of the internet is an excellent additional "reading"
Email hurts me emotionally whenever I have to work it.
ActivityPub and all the new federated social media is people finally realising that having social networks be centralised is a terrible idea! I would really love to see what Dylan's thoughts on it and the networks built on top of it (Mastodon, Lemmy, Pixelfed, etc.) are
IMO Email is a perfect demonstration of why ActivityPub can't accomplish what a lot of folks hope it can. If it gained real popularity, the costs of running servers would quickly outpace the capacity of independent not for profit servers - in fact, it's that's already happened with just the influx from twitter's bullshit, resulting in some servers shutting down entirely while many others closed registrations - and it wouldn't take all that long before a handful of profit driven service providers were hosting the vast majority of accounts, just like we see today with email. And those handful of providers would begin to put up wall after wall to block small self-hosted instances from interacting with their users, using (rightly or wrongly) the specter of spam, harassment and accountability as justification - just as they do with email.
That isn't to say that abandoning siloed social networks in favor of ActivityPub based networks wouldn't be a net positive, merely that the benefits of doing so under capitalism have been vastly overstated by a lot of evangalists. Email would definitely be a lot less useful if it were strictly controlled by a single profit-driven multinational corporation instead of loosely controlled by a handful of them - but it would be vastly better if we got rid of the profit motive that drives those multinational corporations to be so shitty in the first place.
After suckerbergs entry into fediverse with instagram threads, I'm not so sure we won't have Activitypub messed up as well in some time.
In practice, email is a Microsoft/Google duopoly with Yahoo having still a surprisingly large market share.
Matrix is a protocol for federated messaging
i hope someone forks lemmy so it’s not controlled by genocide-deniers, because the idea is good, just the people behind it at the moment are not
"And that's the end of the bit that's going on youtube"
The person uploading this: "Now, that's where you're wrong"
It does end there, it ends exactly there. Was it cut down after the fact?
@@IanBLacy It used to not end there. They must have edited it. I don't remember a thing of what he said after this point though
@@Keisuki Nooooooo 😢 I gotta look for the recordings elsewhere I guess :((
Was kinda curious why nobody's commenting on the cut tho
The guy who said "The net interprets censorship as damage" being universally censored by the Internet at large is hilarious and depressing in equal measure.
This was wonderful. Dylan is becoming our modern day Internet historian. We sorely need talks these that explain why we are at the place we are. At once i am always in awe of how simplicity worked in the early Internet and how we are forced to keep the thing going on the face of threats of all kinds. Thank you.
...God I wish the era of people using _the internet itself_ as social media would come back. Neocities is pretty much carrying the torch by making everyone's little garbage HTML website discoverable.
I cracked up when the screenshot with the dark mode came up!
I been there, creating emails templates, making sure that it looks good on every email client that the company uses and then the QA guy just destroys you because he's using dark mode.
We had to export every asset again with transparency. Fun times!
img { background-color: white; }
@@Dev-sl7fo arghhhhNOOO... it burnss our eyesessss!!
Seriously, I will never understand how people are happy with bright backgrounds. Everything should be dark by default, and then for reading long texts or in bright sunlight one should simply invert the colours and use e-ink displays.
I would love if our national TV dr.dk would enable dark mode. You sit in your cosy living room with dimmed light, select their app, and bam!! 60" of #FFFFFF blinding light with a tiny logo fills the entire room.
@@LeifNelandDk du burde se på NRK i stedet, der er bakgrunnen en fin blå! Bonus: alle snakker mye tydeligere enn på dansk tv...
My first real job was writing HTML marketing emails for American Express, New York Times, and Time Warner at a digital creative agency called Digitas back in 2004. Although I was greatful for the job and they were very nice to work with, writing these emails was the most boring and tedious job I'd ever had. And even then, that HTML was out of date and limited. Certainly interesting.
And when you get an e-mail with html. You directly se it some crap that is not specifically written to you and immediately deletes it.
I’ve been in marketing automation since 2011. Nothing has changed - emails still have to contain bloated, dated html that makes accessibility devices cry. Still doing client specific fukkery because Microsoft.
Dylan is amazing at teaching. I know nothing about this but could follow along and enjoy. Really powerful skill
23:33 small correction, Protonmail has the same feature as Gmail, you can add "aliases" after + sign before @ sign and they all will all effectively redirect mail to the same address
I noted that discrepancy too; "sending" would of course work if he registered "rocket" and then just addressed rocket+whatever, but I suppose the point of the segment was to show that a MAILBOX cannot have a "+" in, and he didn't want to (=strictly, one can claim "shouldn't be forced to-") register just a single word address. Brings up another question: does it still map +x to "Label: x" in the GMail app, and does +x+y+z map to "Label: x", "Label: y", "Label: z"?
Amazing talk through and through, with one remark:
55:40 this is misleading because Gmail is such a behemoth in the mail world that it's starting to setup standards that everyone has to comply with.
Each time I see one of your talks online, I am ever more impressed with just how much of a wealth of knowledge you offer, and I thoroughly enjoy learning from you!
Whatever the subject is, I know I'm gonna have a great time watching your presentations
The number of sites that still reject Gmail plus aliases when setting up accounts is disappointing. For that matter, the "canonical" form of my gmail address has a period in it, and some sites even reject that.
@@techtutorvideos No way you did that, all special characters are removed on gmail. If you have dots on your email, your email is the same without the dot, and as the video explained, the + is used just for alias.
Beautiful spam choir!
Awesome presentation, very lively and informative! 🔥
and this is why corporations are having so much trouble figuring out how to create their own 'metaverse' that only they can own. Because the baseline of all of our primary communication methods has been free for anyone to use.. And they have to somehow compete with that.
As a dev, it would maybe be beneficial to have some standards that finally update and drop the cassette tape. But as an end user, why?
Drop the cassette tape?
@@entx8491 16:25
The other reason they’re having trouble getting a good meta verse is because the concept is inherently flawed
.... no.
Metaverse as a concept is flawed from the beginning...
A centralized service that you interact with through an avatar? Well, I can tell you that shopping online is generally more efficient than walking around a store... Because... I can't search, I can't scroll fast through a list, etc
Metaverse is trying to do the inconveniences of real life and add it to the internet
45:55 "floating point boolean" LOL! I love this guy.
☮️❤️🌈
I love that I jumped when I received an email during the beginning part where he was saying "ok, you're getting an email now".
I hate that I was jump-saddened by Young Matty Perry at 33:54.
Summer of Love was 1967 (nice metaphor though)
Excellent talk. Very entertaining AND very informative.
Amazing talk! Useful advice.
And kudos for reminding me of late 90ies when I had to telnet 25 to quickly check mailserver works.
16:13 I thought he was going to use the keyboard layout as an example. The common staggered qwerty layout is unwieldy and inferior to a non-staggered layout, but we keep the staggering for people who are used to using old typewriters. And those typewriters needed the staggegering because otherwise the hammers would get in each others' way. The staggering is also a case of backwards compatibility reaching insanely far back into the past.
I started my career in the 1990s and did a lot of work with email of various types. At that point, RFCs didn't always carry a lot of weight. People just did what worked for them. Lots of things tended to comply with some of the RFC, but often omitted parts or added proprietary extensions. The original SMTP RFC contained a bunch of dumb stuff (like quotes and comments) that very few people bothered to support. Also, bang paths.
Hey! What's wrong with bang paths? (UUCP admin from the early 90s here ;)
ProtonMail supports plus addressing too, they just call it "+alias". Also they have this feature called "catch-all address" which is a more general version of the same thing, because every address you didn't already register for that domain will be delivered to the catch-all address instead. For example single apostrophe and underscore works just fine, though single dash doesn't.
Great talk! Had a good laugh, and some flashbacks..
The Protonmail *signup* results (not the sending results) make sense if you remember Protonmail is supposed to be some kind of secure service so it probably puts an emphasis on not allowing weird things just in case.
At 28:46 you made a big mistake - you DID ask the Zoho server to relay mail to the Protonmail test domain (pm) instead of sending directly to the Zoho test domain (zh)
Ooops.
@@DylanBeattie what happened after the video cuts? did you talk about Gmail, Azure, AWS, etc? or did you sacrifice someone's first born child?
or was it a round of questions? if the latter, thank you, I hate when uploaded lectures/talks have those xD
Such a good presentation! Interesting and informational!
Hands down my favourite lecture of Dylan. Seemingly obscure, but full of actually fascinating stuff and totally fun.
33:20 "E-mail was invented by hippies. And hippies SUCK at security." XD
Ane awesome talk! Thank you so much, I am very amused and a lot more informed now :)
17:22 Most naive email regexes in Stackoverflow would probably match the first two, sometimes the third and fourth too. But when I when I first considered the problem of validating email addresses I realised that the spec allowed so much the only thing I really could be sure of was that a valid email contains an @ symbol and at least one character before and after it. But of course then the clients complained that it accepted "somename@gmail" etc. without the top level domain; so I decided that it was better to add another technically incorrect requirement that an email should have at least one period with at least one character before and after that period in the host name part; with the assumption that anyone who legitimately have an email without any periods in their host name is probably much rarer than idiots who forgets the top level domain name of their email.
So whenever I have needed to make my own email validation; it would usually accept all of these except the ipv6 one.
I have just tested...
Sending from the work 365 mail I could single-apostrophe-at my personal domain, and... I did get it.
From now on, this is the address I am sharing with people.
Using plus '+' in an email address is not unique to Google. Office 365, Protonmail and Fastmail all support using plus in the email address the same way. (I am not sure about Zoho mail because I have never used it)
Zoho supports plus as well.
It's called "the sendmail hack" because it's that old
Thanks for giving me a good time by watching this!
Excellent talk, Dylan. Thank you.
Hey! You forgot IRC as an open protocol for communication :)
Nice talk, liked it, was very informative. Now I understand raw mail!
One of the best things about this lecture is that I finally understood why I have ALWAYS hated HTML e-mail, and up to this day, forcing plaintext mail is the first thing I do when I set up a mailer. I never really knew why I hated them. But when I heard "table within a table within a table", I knew that even without knowing exactly why, my coder brain just knew it was crap. Thanks, Dylan :)
Really insightful piece. And wonderful delivery
I am always looking forward to the next piece from Dylan.. I'm not even in the tech field but modestly tech savvy and enjoy the humor and education in each one, everything from Rock star programming to the Art of code
Wonderful talk and good history lesson.
Super informative and entertaining!!!
Finally a nice overview on e-mail. Thanks! I set up my own mail server one time but dealing with Microsoft and google was just too much, especially because I didn't know enough about what I was doing.
Great talk, as someone who has to deal with e-mail, troubleshooting clients writing mail templates sometimes It truly is a pain to work with.
Hold on. DNS in 1983? It wasn’t in widespread use. I was a student operator in the digital computer laboratory at the university of Illinois in 1983. At that time our “network” connection was a 9600 baud modem to Purdue twice a day, in which we sent 2 MB or so of email and Usenet postings per connection. Some time later, we got an always on connection of… not sure, 56 kb? Anyway, for years I was typing in all sorts of arcane email paths to send messages. Thank god for ihnp4, which was a gateway to everything! DNS wasn’t adopted for quite some time. The hosts file grew to some 1.2 or maybe 2 MB by the time DNS was rolled out. I was disappointed, because it meant that I wouldn’t be able to look up (grep) for any host I wanted from the hosts file. Of course, it also meant that a single entity couldn’t claim a name that nobody else could use😂. Anyway, I maintain that 1983 was too early for dns to be used. No one was using it, that I was aware of.
Email subaddressing with the plus sign in local part was in use long before Gmail even existed, at least since the late 80s/early 90s by the Andrew system at Carnegie Mellon. Source: I'm old. PS:
Oh, regarding it not being "part of email standard" you are basically right, but as with many Internet standards, it's complicated. RFC 5233 describes subaddress filtering, including "One common way of encoding 'detail' information into the local-part is to add a 'separator character sequence', such as "+", to form a boundary between the 'user' (original local-part) and 'detail' sub-parts of the address, much like the "@" character forms the boundary between the local-part and domain."
Extremely useful info. Thanks!
I guess I need to watch more of his videos, glad I stumbled upon someone talking about him on Reddit.
Plus sign in email addresses is *NOT* a gmail/google specific feature, it is an *EMAIL STANDARD* specified in RFC 5233 (from January 2008) that was originally specified in the now obsolete RFC 3598 (from September 2003)... google/gmail just happens to be one of the big email providers which happened to support it first but more tech savvy people have ben using it for years (on sites which supports it, not all sites consider an email address with a plus sign valid)...
awesome as always!
This talk should be compulsory viewing. OMG. So packed with critical context about how things work in Internet land.
I have no reason to have enjoyed this talk as much as i did. I'm not a developer, programmer, hell i can't remember the last time i opened an email client. But this talk was very interesting.
Amazing!
Rofl, the abrupt ending. "That is where we're going to end the version of this that goes onto youtube*hardcut*
Fascinating!
23:30 Proton mail supports subadressing (that + thing in local part), and office365 does it as well - thus you cannot use + character in standalone email address...
Great talk! Still feel like it should have tried to sell me something because that's the web now.
Not in the places I hang out.
Fantastic talk! So good! :)
As he said "We are now going to send you an email" I got a notification for an email being sent... Just about crapped myself
Best talks in the business, hands-down.
I think plus addressing predates gmail by some years. It is supported in sendmail starting with version 8.7.
Wow this is a great talk 👏
This is one of the best tech talks I’ve seen.
I have ME/CFS. Dark mode allows me to continue to do stuff with computers. Big areas of bright lit pixels like white backgrounds makes me tired faster.
i needed to hear this
I remember manually assembling mime into files, sometimes waiting quite a while for the middle bits of the email to arrive because I was down the FAR end of a long ! path.
I had that moment. Client never got any emails, finding that we changed our mail provider and I forgot to update then smtp client details on our backend lol
This is an amazingly interesting talk.
I'm confused by the title email vs capitalism. Is it because capitalism caused the spam? or was there something else I missed?
13:42 OK we all, the entire internet formally agree to never send spam. Amen.
2023 what absurd big number of bytes spam per second ?
light mode is what shouldn't have been allowed in the first place. a monitor isn't a piece of paper dammit
Great talk as usual @DylanBeattie
One thing - Microsoft have the same +tag tracking on 365 23:26
You know a talk is truly excellent and the presenter is an engaging speaker when you're already familiar with all of the concepts being communicated but you still watch the full hour and love every minute of it. Amazing work, Dylan!
38:50 Such a powerful quote given the ad-blocker ban situation in CZcams.
I've seen a few providers support the + tagging method, it's a very useful feature I wish was made standard.
I vaguely want to know what happened after the youtube version ended…
Awesome presentation! On the last point about corporations owning WhatsApp, Telegram, etc, could the ActivityHub be the email equivalent for modern communication needs?
I used to enjoy talks from people like this, but that was then.
Great talk.
The + character in the local part of an address seems to have been described in the 2008 RFC 5233.
49:00 seriously considering parsing HTML e-mail because looking through that crap gets tedious.
I have been resisting HTML e-mail because I feel it weakens any security guarantees of signed messages (since different clients render HTML differently).
HTML mails were/are huge mistake... Usually doesn't even deliver on the look-good side, and ruined so much stuff. Just because some people think using few extra asterisk and ascii-art tables is NOT enough to emphasise text. :(
(also my email client renders them in a way you wouldn't expect if you haven't been using text terminal ever in your life... yes, I'm reading my emails in text mode with alpine email client and html mails are sort of stripped down to text-only form for me, in nice green font over black background of text terminal)
@@ped7g When I was still on dial-up lynx was my favorite browser. It was like 5x faster than any other browser because it was not downloading superfluous crap.
I finally moved to graphical browsers in the 2010's.
No, Non-dark mode on any site/program/app/screen thingy should be illegal with capital punishments.
Should be as default dark mode
Noooo I need to know what happened after the live stream. I need closure on the anecdote.
I actually still remember my first spam mail in around 1996. Some guy from Nigeria telling me in broken English that I should send $$$ somewhere to get rich. I was just horribly confused. Do I know this guy? Why does he know me? Am I danger? May I actually really get rich?? And especially, why is his English SO incredibly bad as to being nearly undecipherable? Sounds strange today where everyone has built-in spam filters in their brain after a few months of e-mail usage... but that's how I remember it :)
Love the Bobson Dugnutt reference at 20:21
Fastmail is highly recommended! Simple, no-bullshit, to the point.
At 36:16, you said that in the original SMTP spec, methods could have been used to better track senders and include spam protection in the standard itself. Do you have examples of such methods or ideas that could have been implemented ?
Listen I'm not saying I was bored, because I loved the talk up to the point that it happened, but the giant spike in the graph of my enjoyment over time when you made the Vision joke was amazing.
Fabulos talk!
I watched to the end, now I have to know... what was it that didn't come on to youtube????
Thank you, that was very interesting. I thought you might touch on pgp and gnupg and why nobody ever mandated that system (as far as I'm aware).
I love dark mode on email, I guess I'm a devil! (joking)
I loved the talk, everything was explained in a manner I, someone with minimal background in this topic, could understand.
Wish there was more mentions about how email was/is at odds with capitalism given the video title, but maybe that was clickbait and I got played.
The IMMEDIATE cutoff at the end left me laughing my ass off!!!
38:12 if you also wanted to know what it was that is base64-encoded there: it is "funwith:email", so a colon between login and password.
About 53:46 : Dark mode is the original default *and* an accessibility feature. It's bright mode that should never have been allowed to happen.
This was really interesting!