@Colm @stanch I will be really thankful if you can share some final thoughts on this. It will help me move in the best direction possible as of now.
I have started a parallel topic for better visibility.
Link: Writing Custom Spark Lake Loader for Iceberg
Thank you!!