- 15
- 19 476
Duane Bester
United States
Registrace 17. 09. 2010
1 BILLION row challenge in Go - 2.5 Seconds!
In this video we look at how we can aggregate 1 billion rows of weather station data in as little time as possible. We start with a naive approach and optimize to go from 1m30 seconds to 2.5 seconds. Using the power of memory mapped files, a custom hash map implementation, and multiple Goroutines.
Implementation: github.com/duanebester/1brc-go
Thanks to Ben Hoyt
benhoyt.com/writings/go-1brc/
00:00 Intro
00:54 Simple Implementation
09:55 Advanced - Using mmap
15:01 Custom integer parsing
22:12 Parallel processing
34:16 Custom hashmap
42:48 Results
Implementation: github.com/duanebester/1brc-go
Thanks to Ben Hoyt
benhoyt.com/writings/go-1brc/
00:00 Intro
00:54 Simple Implementation
09:55 Advanced - Using mmap
15:01 Custom integer parsing
22:12 Parallel processing
34:16 Custom hashmap
42:48 Results
zhlédnutí: 8 324
Video
Simple list of items with GPUI - The native, high performance, GPU rendered Rust UI framework
zhlédnutí 731Před 2 měsíci
In this stream we look at building a simple list with GPUI. We want to be able to append items and scroll up and down to view all the items. Github source code for list examples - github.com/duanebester/gpui-list
Bidirectional communication in GPUI with Web-sockets!
zhlédnutí 381Před 2 měsíci
In this stream we improve on previous code from the last video. We find our way towards being able to send and receive messages in GPUI via Web-sockets! Github repo with code - github.com/duanebester/gpui-websocket 00:00 Intro 01:00 Workspace 03:43 Message view 05:21 Update view 10:38 Break up Chat 18:11 Subscribers 19:46 Handle button click 24:55 Web-socket connection 27:16 Heartbeat 29:12 Sec...
Second look at the Rust GPUI framework from Zed! Featuring web-sockets
zhlédnutí 1,5KPřed 3 měsíci
In this stream we look at how we can get messages from a web-socket connection running in a background process, to update a view in GPUI. GPUI - gpui.rs Zed - zed.dev Github link with the project files - github.com/duanebester/gpui-websocket 00:00 Intro 01:15 New cargo workspace 02:57 Hello world up and running 05:15 Multiple apps 13:26 build_view and build_model 14:30 Background task 19:43 Bac...
First look at the new Rust GPUI framework from Zed!
zhlédnutí 4,1KPřed 3 měsíci
In this stream, we continue on our Rust learning journey by exploring the newly open sourced GPUI framework from Zed Industries. The framework allows us to build UIs that are rendered via the GPU for high performance. 00:00 Intro 00:29 New Cargo project 01:20 The big picture 03:43 Hello world 05:21 Theming 07:31 Could not compile issue 10:00 Theme names 12:20 UI::Button 15:00 Counter 15:21 Butt...
Async Rust called from Python!
zhlédnutí 95Před 5 měsíci
In this tutorial we look at how we can take an Async Rust library (calls starwars api via OpenAPI generator) and use it in other languages (Python) via the UniFFI tool! Github Repo: github.com/duanebester/rust-ffi-starwars-api 00:00 Intro 00:43 New Rust Library 01:33 SWAPI and OpenAPI 04:31 Generate Rust Client with OpenAPI 08:00 Creating the Rust Library 10:00 Testing the Http Request 12:08 Ad...
Codegen a Go Fiber app with OpenAPI
zhlédnutí 2,4KPřed 6 měsíci
In this video we look at how to build an API with an API-first approach. We leverage oapi-codegen to generate a lot of boiler plate for a Go Fiber app.
Building an OAuth2 Server in Go - Part 3
zhlédnutí 170Před 6 měsíci
Part 3 of a few streams, we set out to try build an OAuth2 server from scratch to better our understanding. We will make mistakes along the way, but every step is a step forward. In this stream we add endpoints for creating users, and a new service to received the callback details. With these details we can request an access token. Twitch: twitch.tv/duanebester
Building an OAuth2 Server in Go - Part 2
zhlédnutí 236Před 7 měsíci
Part 2 of a few streams, we set out to try build an OAuth2 server from scratch to better our understanding. We will make mistakes along the way, but every step is a step forward. In this stream we implement the token endpoint for generating an access token. 00:00 Recap 01:55 Start our /token endpoint 03:08 TokenRequest model 14:22 Gorm null string 17:40 Gorm update model 26:06 Copy over JWT cre...
Building an OAuth2 Server in Go - Part 1
zhlédnutí 927Před 7 měsíci
One of a few streams, we set out to try and build and OAuth2 server from scratch to better our understanding. We will make mistakes along the way, but every step is a step forward. 00:00 Docker and dependencies 08:49 main.go 09:51 godotenv 12:07 Coding our Client model 22:28 Running basic API 24:30 Building AuthRequest model 26:43 Building /auth endpoint 36:07 Setting up a cookie 37:07 Setting ...
Deploying a Go Fiber API to Fly.io with Database!
zhlédnutí 188Před 7 měsíci
In this stream we build and deploy a simple Golang Fiber API to fly.io with database. So fast and easy! 00:00 Simple Fiber API 00:20 Install Flyctl 01:20 Basic Dockerfile 02:11 Install GORM 03:00 Setup ENV variable for database url 03:57 Add new User model 05:19 Add new user endpoint 06:17 Remove dependency on database 06:50 Deploy docker file to fly.io 09:03 Deployment Errors 09:24 Fix dockerf...
Refactoring Go API - Breaking out services
zhlédnutí 124Před 7 měsíci
In today's stream we look at one potential way to break out our Services and API Handlers. My Twitch: www.twitch.tv/duanebester Source Code: github.com/duanebester/go-series-app
Adding JWT Authentication to a Go Fiber app
zhlédnutí 247Před 7 měsíci
In today's stream we look at some of the Go Fiber examples and add JWT based authentication to our small example API. Twitch: www.twitch.tv/duanebester Source Code: github.com/duanebester/go-series-app Go Stream Series Playlist: czcams.com/video/g1fl41OewQA/video.html
Adding ENV config and hot reloading to Go apps
zhlédnutí 55Před 7 měsíci
Suffered a bit with GOROOT and GOPATH, but we made it through! Added environment variable config with Viper and hot reloading with Air Twitch: www.twitch.tv/duanebester Source Code: github.com/duanebester/go-series-app Go Stream Series Playlist: czcams.com/video/g1fl41OewQA/video.html
Building a Go app with Cobra and Fiber
zhlédnutí 159Před 7 měsíci
In today's stream we look at building an app with Go Fiber, and then decouple certain functionality via Cobra commands! This allows us much more flexibility in the long term. Twitch: www.twitch.tv/duanebester Source Code: github.com/duanebester/go-series-app Go Stream Series Playlist: czcams.com/video/g1fl41OewQA/video.html
keep distance from the mic. don't over process the audio. Just speak,sound like normal human please.
Thanks- This has been better in more recent videos
Thanks for sharing.
You bet
Have you tried to use more goroutines than you have cores? And possibly playing with GOMAXPROCS? I suspect it isn't going to make any positive difference, but I'd try it. (Back in the day when I worked on high throughput systems and we used spinning disks, one secret was to always keep the (SCSI) disk operation pipelines full so the firmware could do reordering on the operation queue to optimize performance. The easiest way to do that was to provide the kernel with lots and lots of runnable processes. So the load on those machines was always on the order of 2-300. Which sounds bad, but we didn't care about latency - only throughput. I can't remember how much faster this was, but it was significant enough that we ran like this for several years. Anyway. These days everyone uses SSDs and I know nothing about modern disk systems. So this direction may not have any benefits today. But I'm curious what the effect would be :-))
I tried doubling the goroutines, but didn’t see too much of an impact :( I switched to a slightly different hash implementation and was able to shave about 200ms off of the above which was cool. I suspect there are still optimizations to be had by using pointers and non branching code. I’m looking at a C implementation for inspiration here. I can run a C impl in about 1.9sec. Thinking about posting a video on it as well.
It takes me 1 minute just to cat the file to /dev/null
Yep the aggregation calculations are what increases the time drastically
Great content, but the autopilot is taking the fun out of it
Great point. Will disable going forward!
Nice video. However, you are calculating the average while you are meant to keep track of the mean of all the values. That means having an array in the struct to keep track of all the values seen. Subscribed!
I think “mean” in this case is the arithmetic mean, which is the same as the average; summing the numbers in the set and dividing by total count (per station). My output matches the baseline output so I feel pretty confident in the implementation
Hello Duane! This is some amazing stuff man! Keep making these and enlightening us! Thanks a lot!
Thanks, will do!
As always, banger video! 🤤
You already know!
Thank you for blessing us🙏
Any time
Hey cool content!!, i was just following through why did u stored temp code in cookie on initial request, that blog from aaronparecki says to create it only when the user allows the app request, umm not sure which is right now.
Hey there! I can’t remember, but that might change in the 2nd and 3rd videos. There are many ways to store a code temporarily and this was a quick way to do so at the time. However, it’s worth noting that the cookie can be set to secure https only with a timeout and so their shouldn’t be much security concern if implemented correctly
Amazing brother! We should collaborate!
Yeah for sure
Yeah for sure
Thank you for the knowledge brother!
Any time!
Great work, very helpful in getting to grips with GPUI. I've done a lot of work with egui and I like it for rapid development, but I feel like GPUI is very promising. Especially since it's not strictly immediate mode.
Glad it helped!
@Duane - is there a Github link ?
Sorry for my late reply - github.com/duanebester/go-oauth2
Rust lurker here. What are those Div trailers in several lines of the render method?
the div().flex().text_xl() stuff? Part of their tailwind-style layout engine. I believe it's powered by taffy - github.com/DioxusLabs/taffy
Thank you for sharing.
Of course! Glad you’re watching
Thanks for your work this is really helpful!
Thanks- Glad it’s helpful!
great series!
Thanks!
awsome sir there is with golang?
GPUI is a custom UI framework built in Rust- not sure of any similar options for Golang
Great! 💎 Looking forward for the next episode! One thing would be also great to have as an example: scrollable view.
True- good idea
Thank you, great introduction for gpui 👍🏻
Glad you liked it!
For some reason, I cannot run oapi-codegen in the terminal after installing it with "go install".
Have you added your GOPATH to your PATH env ?
Great video! How do you apply a middleware to only some handler functions?
One approach is using route groups - example here: publicRoute := app.Group("/public") apiRoute := app.Group("/api", middleware.Protected(), middleware.EnrichUser(services.UserService)) Full app - github.com/duanebester/dermsnap-api/blob/main/app/app.go
1:36, you can run `cli: install` from the command palette, and then in the future, open directories from the terminal via `zed .`
Ahh yes, thank you!
Why don't you use zed editor to write gpui demos?
I’ve been trying it, VSCode is still my comfort for streaming
your audio is doubled
Thanks will look into this!
Thanks for checking out gpui! It was super cool you were able to pull in stuff from the theme and ui crates and get them working easily :)
Thanks for working on Zed and making something awesome. Looking forward to using/ learning more. I did hit that weird borrow error from the storage settings around 8 min in. Not sure if anyone else hit that
@@duanebester of course! We are happy the community can finally engage with it! I’ll get the team to take a peek at that error.
Quality stuff. Very much appreciated!
Thank you!