How long did the enrich step take? We’ve found they should be similar in time.
I believe we’ve seen both the enrich and shred steps hang when trying to process larger batches and not trying to optimise the cluster setup. I created a thread this morning here.
If you look around the Application History and Monitoring tabs do you see any signs it might have hung? You seem to be access most of the application logs through Application History. Unfortunately I don’t have enough Spark experience to say what you might be looking for.
Gareth