Experimenting with Distributed Data Processing in Haskell • Utku Demir • YOW! 2019

This presentation was recorded at YOW! 2019. #GOTOcon #YOW
yowcon.com
Utku Demir - Software Engineer at Movio
RESOURCES
mastodon.nz/@utdemir
/ utdemir
/ utdemir
github.com/utdemir
utdemir.com
ABSTRACT
#ApacheSpark is one of the most popular data processing frameworks in the world, and is widely used in the enterprise. Its popularity is due in no small part to its adoption of the functional paradigm: it demonstrates the advantages of purity, higher-order functions and laziness to simplify the processing of large datasets. #Haskell excels at all of those things; so it is only natural to think that Haskell would be a good fit for distributed data processing. #Tweag.io's Sparkle and #Soostone's Hadron are a few examples in the Haskell ecosystem.
'distributed-dataset'[1] is a framework written in Haskell designed to efficiently process large amount of data. With the StaticPointers extension of #GHC Haskell, we are able to distribute a computation across different machines; and using the technique described by Matei et al[2] that led to Apache Spark, we can express and execute large scale data transforms using a pretty DSL.
In this talk, I am going to give a brief introduction to the library, and then move on to explaining the key implementation ideas and the advantages that Haskell offers to distributed data processing. [...]
RECOMMENDED BOOKS
Rebecca Skinner • Effective Haskell • amzn.to/3SxTpwY
Vitaly Bragilevsky • Haskell in Depth • amzn.to/3EXpmbeSaša Jurić • Elixir in Action • amzn.to/2RZh5eN
Dave Thomas • Programming Elixir ≥ 1.6: Functional • amzn.to/34Dw3O5
McCord, Tate & Valim • Programming Phoenix 1.4 • amzn.to/3zcUqj4
/ gotocon
/ goto-
/ gotoconferences
#DistributedDataProcessing #DataProcessing #Programming #ProgrammingLanguage #SoftwareEngineering #FP #FunctionalProgramming #UtkuDemir #YOWcon
Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at gotopia.tech
Sign up for updates and specials at gotopia.tech/newsletter
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
czcams.com/users/GotoConf...