IMHO - not enough focus is put onto offline batch predictions. Surely online models are important - but offline models is where most companies start (I imagine)!
a l that’s totally reasonable. With batch prediction though you can normally download the model and call the serving code directly. In Spark especially batch predict is much easier. But certainly it’s a bit of manual work to accomplish.
@@HoldenKarau well, yeah, you can download and run it manually once. but I'm talking ml (with retraining) being part of an etl. for instance you load you data into your warehouse, and one or several columns come from ML (freshly trained on every batch - say daily) - that would seem like a more reasonable use-case to start with for many companies. Rather then serving model online (which is useful too). Like integrating ML into something like Airflow
![not so mini: [SPARK-47672][SQL] Avoid double eval from filter pushDown -- rewrite to decrease num…](http://i.ytimg.com/vi/WHb7St5beBs/mqdefault.jpg)
IMHO - not enough focus is put onto offline batch predictions. Surely online models are important - but offline models is where most companies start (I imagine)!
a l that’s totally reasonable. With batch prediction though you can normally download the model and call the serving code directly. In Spark especially batch predict is much easier. But certainly it’s a bit of manual work to accomplish.
@@HoldenKarau well, yeah, you can download and run it manually once. but I'm talking ml (with retraining) being part of an etl. for instance you load you data into your warehouse, and one or several columns come from ML (freshly trained on every batch - say daily) - that would seem like a more reasonable use-case to start with for many companies. Rather then serving model online (which is useful too). Like integrating ML into something like Airflow
a l totally. Inside of Kubeflow there’s a pipeline system one can use, but in the current 0.4 release has some hard GCP dependencies.
@@HoldenKarau is there a tutorial for that? could you drop a link? GCP lock-in is NOT an issue - going with GCP either way!
a l that’s great. I don’t know if a tutorial for it yet, but I’m certain one will exist soon if not already