View Categories

What is handling large datasets during transformation?

Parthiban Kannan
Updated on February 11, 2026

< 1 min read

  • Handling large datasets means designing transformations so refresh stays fast and doesn’t crash memory.
  • I always filter required date range early to reduce rows pulled from source.
  • I remove unused columns before joins to minimize processing size.
  • I rely on query folding so database does heavy work instead of Power BI.
  • For huge tables, I implement incremental refresh instead of full refresh.
  • I avoid row-by-row custom columns and prefer aggregations at source level.
  • Example: Reduced 25M row dataset by pre-aggregating daily sales in SQL view.
  • I also disable auto date/time and unnecessary type detection.
  • Goal is stable refresh, lower RAM usage, and faster report load.
Scroll to Top