https://amuraru.github.io/index.html Recent content on @amuraru Hugo -- gohugo.io en-us <a href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank" rel="noopener">CC BY-NC 4.0</a> pe="application/rss+xml" /> https://amuraru.github.io/about/ Sat, 25 May 2019 00:15:06 +0300 https://amuraru.github.io/about/ Adrian Muraru @adobe Open Source enthusiast https://amuraru.github.io/posts/2019/05/spark-rdd-shuffle-internals/ Fri, 24 May 2019 23:45:10 +0300 https://amuraru.github.io/posts/2019/05/spark-rdd-shuffle-internals/ group-by is using an aggregator to combineValuesByKey At the DAG level configuration: See PairRDDFunctions.scala#L503-L505 val createCombiner = (v: V) =&gt; CompactBuffer(v) val mergeValue = (buf: CompactBuffer[V], v: V) =&gt; buf += v val mergeCombiners = (c1: CompactBuffer[V], c2: CompactBuffer[V]) =&gt; c1 ++= c2 which eventually calls: PairRDDFunctions.combineByKeyWithClassTag val aggregator = new Aggregator[K, V, C]( self.context.clean(createCombiner), self.context.clean(mergeValue), self.context.clean(mergeCombiners)) which in turn, boils down to: new ShuffledRDD[K, V, C](self, partitioner) .setSerializer(serializer) .