Spark Fundamentals — Screening
A phone-screen round on how Spark actually executes work. Answer from first principles.
What is the difference between a transformation and an action in Spark?
Where does the driver run, and what is it responsible for?
Which operation forces a shuffle (data movement across the network)?
Why is calling df.collect() on a large DataFrame dangerous?
What is data skew and why does it hurt?
A broadcast join is appropriate when...