The content of a document to be indexed can grow infinitely. We need to set a limit. Large Excel can be as 150MB, this buffer will be copied and than indexed which is very memory intensive.
- There is a maxFileSize parameter to limit the maximum size of files, use it as the max. value for the content.
- The index writer is accessed from different threads, the main indexer thread and the different pooled folder indexer, fix this, it's illegal from Lucene point of view.
- The folder index pool create zillions of threads and it's not really pooled, use a thread executor instead which is really pooled.