Remove Outliers — Solution & ExplanationProblems / Remove Outliers / EditorialApproachOutlier removal using standard deviation is a common data cleaning technique.SQLUse scalar subqueries to compute the mean and standard deviation, then filter with BETWEEN:```sqlWHERE value BETWEEN(SELECT AVG(value) - 2 * STDDEV(value) FROM measurements)AND(SELECT AVG(value) + 2 * STDDEV(value) FROM measurements)```PySparkAggregate the statistics into a one-row DataFrame, then crossJoin it with the original data to make the thresholds available for filtering.Note: STDDEV computes the sample standard deviation (divides by N-1), not the population standard deviation.Hints

Use subqueries to compute AVG(value) and STDDEV(value) from the full table.

SQL's BETWEEN operator is inclusive on both ends, which matches the ≤ requirement.

In PySpark, compute aggregate stats as a single-row DataFrame and use crossJoin to combine with the original data.

Solve Remove Outliers yourself →