Large files, multiple destinations

[rosebaby3892]

When using SFTP you can transport large files to a single destination without ever loading it in memory. Simply turn on the stream download and you are good to go. For one of my customers I had to create a route that does the same but then to multiple destinations. With files going upto 45 GB this could not be done in memory and just streaming it wont work because you can only read it once.

When sending a stream to multiple destinations with camel you need to special lead enable streamCaching. This “caches” the stream and allows you to access it multiple times. Normally you simply add the streamCaching option to your route and it works. However when doing this with a large file I got an out of memory exception indicating it couldn't transform my string to a streamCache. Now my perception was (as it is stated in the documentation) that if a stream is to large it is offloaded to disk automatically by the streamCache implementation. Even when I changed the thresholds to 1kb.

After some debugging I noticed that this option was never triggered. Some checks in an if statement where preventing it. After some tinkering I came up with the following logic / configuration that enabled the offloading to disk:

?
private void configureStreamCaching(){
    StreamCachingStrategy streamCachingStrategy = new DefaultStreamCachingStrategy();
    streamCachingStrategy.setSpoolThreshold(spoolThreshold);
    if (!spoolDirectory.isEmpty()){
        streamCachingStrategy.setSpoolDirectory(new File(spoolDirectory));
    }
    streamCachingStrategy.setEnabled(true);
    streamCachingStrategy.addSpoolRule(length -> spoolThreshold < length);
    streamCachingStrategy.setSpoolChiper("AES/CTR/NoPadding");
    getContext().setStreamCachingStrategy(streamCachingStrategy);
}
 
<span class="copy">Copy</span>
I had to add the spoolRule and I had to enable the strategy. The cipher is set so that the tmp files are encrypted, you've got to keep your security officer satisfied! Now I also wanted to be able to determine the directory. The “strange” thing that I had to do was that I had to set the spoolDirectory as a File object. You have the option to set it as a String. However, this only works when the strategy is not yet started. Using the File object ensures that the directory is actually used once set, no matter when you do it. A final note on this is that you have to make sure you have enough disk size, because that will be the limit of the file size you are moving.