Meta (Facebook) Machine Learning Mock Interview: Illegal Items Detection
VloĆŸit
- Äas pĆidĂĄn 17. 01. 2022
- Today Zarrar talks us through this question asked by Facebook about how to use Machine Learning to flag illegal items posted on a marketplace.
Try adding your own solution to the question here: www.interviewquery.com/questi...
đ Subscribe to my data science channel: bit.ly/2xYkyUM
đ„ Get 10% off machine learning interview prep: www.interviewquery.com/pricin...
â Check out our machine learning interview course: www.interviewquery.com/course...
đ Get professional coaching from Zarrar here: www.interviewquery.com/coachi...
đŠ Follow us on Twitter: / interview_query
Attention Hiring Managers & Recruiters: Ready to find the top 1% of machine learning talent for your team? Accelerate your hiring with Outsearch.ai. Their AI-powered platform seamlessly filters the best candidates, making building your dream team easier than ever: www.outsearch.ai/?...
More from Jay:
Read my personal blog: datastream.substack.com/
Follow me on Linkedin: / jay-feng-ab66b049
Find me on Twitter: / datasciencejay
Related Links:
Facebook Data Science Interview Questions: www.interviewquery.com/blog-f...
Facebook Data Science Internships: How to Land the Job: www.interviewquery.com/p/face... - VÄda a technologie
This guy has mastered the art of how to talk for 20 minutes something that can be explained to a technologist in 2 minutes, and that my friends is what a system design interview is all about. You have to talk about every detail no matter how boring/mundane it is to you or how obvious you might think it is.
Typical end to end ML Question:
Understate the problem, Data collection, Feature Engineering, Building Model, Train Model, Evaluate Performance ( Confusion Matrix: Precision ± Recall) , Deploy Model, Rebuild Model if needed
Good summary!
Decent summary, but most FAANG interviewers would probably dock for not discussing online training, A/B testing, exploratory analysis for selecting model
I feel this video is a fantastic resource, not only the explanation was great and very insightful, but I think you also made the right questions, going for the extra-mile of the explanation/analysis...thank you for sharing!
Wow this guy is good. I really like how he start from model framework with baseline model, point out the reasoning and key considerations - and we can evolve from there to more complicated model just by all similar reasoning
Amazing! As a point to improve even more, Iâd add as finishing touch fine-tuning the model with adversarial examples.
2 points that I would added for the end questions:
1. in order to overcome the coded firearm words -> use tranformers models like BERT as you can catch the meaning by the embeddings (ie: cosine similarity) and filter the best ratings
2. Computer Vision on the images can be used as additional inference if the F1 score is low, but not always as this type of inference is more expensive
This is a fantastic video for giving an idea for an ML system design interview ! Thanks for making this.
Should've mentioned that people try to disguise the actual product description using proxy words.
Also, to include image analysis or not, I'd draw multiple samples and train models in A/B setting. Then run a t-test to see if the mean prediction metric is significantly different or not.
Were you use white board for ML design architecture? Is white boarding helpful in the interview?
Thanks for tuning in! If you're interested in learning more about machine learning, be sure to check out our machine learning course. It's designed to help you master the key concepts and skills needed to excel in machine-learning roles.
www.interviewquery.com/learning-paths/modeling-and-machine-learning
It's also possible to use re-ranking or bagging approaches to combine xgboost model and vision/nlp model, which would most likely improve performance
you mean use a gradient boosted tree in the first stage and in the second stage use a vision/mlp model (which is more complex and takes longer to excute)?
Great Interview Zarrar!
You wan a system that overfits and hits lots of false positives, as false negatives can be catastrophic, legally, for the reputation of the business, could even lead to regulatory action and media scrutiny, killing sales, market cap, etc.. You then have agents go through the false positives and efficiently decide if they are truly false positive or not. This data can also help train the model. The cost of hiring people to go through and check is much lower than losing 5% of market cap due to negative press.
Great videos! Where do you get the sample questions from shown at the start of the video?
www.interviewquery.com/
what does the following mean? TF-IDF: "We scale the values of each word based of each frequency in different postings"?
Sorry, where did you discuss the label generation part? There are multiple ways to generate labels with pros and cons:
1. user feedback: Automatic, lot of data but noisy.
2. Manual annotation: accurate labels but not scalable. Very high proportion of examples would be tagged as negative.
3. Bootstrap: Train a simple model and sample more examples based on model scores to get a higher proportion of positive examples.
4. Hybrid: Manually annotate examples marked as "X" by users where "X" can be tags like "illegal", "offsensive", etc.
You can also scrape for images, and generate listing using LLMs for high quality synthetic data.
Re; whether or not to do CV on images - shouldn't one do error analysis to check if text and other features lacked the predictive power and the signal was elsewhere (aka images) which is why we should invest in extracting signals from images; as opposed to building a giant model with all features and doing ablations to understand feature class importance. Latter seems quite expensive?
If you are working for FB, you can afford to go for an expensive model. If a candidate didnât mention CV, I would be unhappy since there is a good source of data you are not making use of.
Great insights to sample questions
great insights but the text data can be various language but when he also said augment the some keywords to detect can that work or train different language different??just curious
Synonyms and similar words can help embellish the classifier and create new features
Hello. I think this was super helpful overall. I'm a little confused when he describes Gradient Boosting. For each successor tree, we should set new target labels for training errors in the predecessor, no? (and leave the weights alone)
I think he was talking about adaboost instead of gradient boosting.
@@jiahuili2133 how does a Gradient Boosted Tree work in this context? Any other models would could use here? Unsupervised machine learning?
â@@Gerald-iz7mvYou already have labels, though. So supervised learning is probably superior.
@@sophiophile but labeling the data is a lot of effort?
@@Gerald-iz7mv You already have labelled data in this case. They described having the historical set of previously flagged posts. Also, expecting to cluster out the gun posts in an unsupervised manner when they make up such a small proportion of the listings is unrealistic. The other thing is that feature engineering and labeling pipelines are simply part of the job, when it comes to ML.
Nowadays, you can also very easily create synthetic labelled data of a very high quality using generative models as well to help with the imbalanced set.
Wow this was so useful.
Around @12:00 the algorithm that upweights incorrect prediction is Adaboost instead of GBM, right?
Yes.
I can never remember what Precision and Recall stands for. It is clearly visible how the interveiwee was also confused and video is edited around that point.
thanks Zarrar
this is the best video ever
Great video. I find all the quick cuts to be a bit disorienting though.
I would have suggested CNN as an alternative approach but ya agree. The listing is not only about an image but also text. Edge case where they have different text and different images then that won't get captured. Thank you.
I haven't watched the video yet, but a lot of people will dock points for over-engineering. I haven't seen his suggested solution yet, but if a really basic ensemble approach (one model for image, one for text) can achieve the goal instead of a single multi-modal one and with less resources at every step- go for that and explain why.
Now, to be fair, you were commenting prior to the multimodal LLMs being everywhere, so that does change the considerations.
@@sophiophile thank you
Very useful! Thanks for sharing. Do they ask about data pipelines and technologies that might be useful to scale the model (for the MLE role)? Would love to know more resources on it! as well as more mock interviews :)
Definitely in the MLE interview loops!
INFORMATIVE GOOD SIR
@iqjayfeng I think Zarrar mistakenly mixed up False Pos and False Neg around 2:00 mark. It would be ok if customer service received False Neg (model pred True but its really False) not False Pos
Excellent
Thank you! Cheers!
Why does he say that it is a better idea to use NN rather than gradient boosted trees if we need to continuously train/update the model with every new training label that we collect from the customer labeling team?
Because you can update a NN weights with just new data points by fine-tuning unlike tree based models which *may* require re-training with old+new data
Remember tree based models are sensitive to change in data
GBM is fast to train?????
F2 score will be better here I think đ€
If the dataset is biased? Why bother using accuracy as the metrics to evaluate the model?
I believe he was using accuracy in its semantic meaning, not the actual metric. He already said he would use F1, and then referred to it as âaccuracyâ because itâs an easier word. Probably âscoreâ would have cleared the confusion.
Too mock, not like real interview, all things were mouth work without any drawing and writing.
intellectual masturbation