Processing data sources with large number of fields(columns)
When processing data sources with many fields or columns (e.g. 200 and more), you might encounter the following error when running a job:
Code: Select all
Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
To overcome this, please increase the available memory for the profiling job via the run configuration of the job (accessed via the Profiler > Run Configurations menu, see attached screenshots). For example, 250 fields usually require around 600MB of memory.