Shouldn't Data Connections Be Easier? (with Ashley Jeffs)
Vložit
- čas přidán 23. 01. 2024
- Benthos wants to be part of your Data Engineering toolkit - it’s there as a quick and easy way to set up data pipelines and start streaming data out of A and into B. In contrast to a lot of the tools we’ve talked about on Developer Voices, Benthos seems focussed on cutting development time down to a minimum, so you can quickly configure a new pipeline and test it out, without making a whole sprint of the task. As quick as a quick-and-dirty shell script, without the dirt. 😉
So this week we’re talking to the creator of Benthos, Ashley Jeffs, to hear why he created Benthos, what it can do for you, and what its strengths and weaknesses are. And Jeff’s refreshingly candid about when you should and shouldn’t use it. If you ever need to get data from an HTTP connection into S3, or S3 into Kafka, or Kafka into a flat file, Benthos might just save you a few hours of development.
-
Benthos: www.benthos.dev/
A list of supported inputs, processors & outputs: www.benthos.dev/docs/about#co...
All the cute blobfish logos: www.benthos.dev/blobfish/
IDML: idml.io/
Kris on Twitter: / krisajenkins
Kris on LinkedIn: / krisjenkins
-
#software #podcast #dataengineering #datascience
Oh boy, my company is going to use the hell out of this, even if they don’t know it yet
Thank you for these great interviews. I'm always surprised how HARD things are. I work with financial data and JS likes floats, accounting likes fixed point or ints to the cent. And a lot of times I'm sending CSV files with text that will be converted to whatever type is used by the receiving database. Data Pipelines are HARD. Sure, Debits and Credits work with INTs, but what if I want to multiply by a monthly interest rate to 5 significant digits. A lot of times I'm not looking for "correct" calculations, but consitant ones. I love the precept of "LOVE YOU DATA." And always have to beware things that work perfectly fine for 90% of cases. That 10% gets you in trouble.
Just excellent, thank you both!
I like it and may be will use it for personal projects first, i guess.
About writing custom inputs and outputs @45:30 it was said: "the main function is 3 lines (of Go code)".
It sounds like trivializing the problem. It probably comes from the biased view-point of Benthos being written in Go.
From another point of view, when u have a system written in ANY other language, eg. Clojure, then a custom plugin is not a small thing at all.
U would need to pull in hundreds of megabytes of the Go eco-system, not only to the developers' machines but to CI servers too.
Plus now you have to revert back to the edit-compile-run a whole program paradigm, which is magnitudes less pleasant than lisps' single expression edit-run REPL experience.
(I understand, as it was mentioned, that using a sub-process can circumvent this problem in many cases, which is great, despite it's just handled as a byte stream.)
I've had my eye on Benthos for a little while. It looks really good but I haven't had a use-case yet.
see anything i hear something like this ...the only thing that is important fault tolerance, recovery and volume it can handle. these are important for any data tool to float....how precisely you can say the problem statement is marker that you're solving something valuable or may be arrived target product-market fit. And on top of it if you can show/demo it like a magician then at that point you're sitting on top of cashflow.
@@jessepowellr4 explain yourself??
chapters would have been nicer to have in this video.