This is my old posting posted at planetsourcecode.com which I am re-posting on my own blog...
Recently, I got an assignment to create a module for archiving the transaction files. The files range from 1KB to 100GB. The system gets about a million such files everyday for transactions generated for Asia pacific region from Citibank customers all over the world. The destination can be any type of system (windows, unix, mainframe, etc).
Good thing for us is that these files can be identified by its sources and time zone. The best part is that 100GB files comes from US, they are consolidated files and max 12 a day. I need to focus on the Japan firm banking system since they have multiple 1KB files and all need to process immediately.
Unfortunately, there is no Java API to do it and we cannot use 3rd part tools. I have written such solutions in Unix shell scripts earlier too, hence the concept is clear.
Solution:
1. Java component to go round-robin to poll the incoming folders (one for each destination) to look for files. This component will manage threads for using existing NMD licenses of Citibank (configurable in XML file).
2. Java thread component to create threads using the priority set in configuration for each destination folder.
3. Java component to write the Unix shell script and execute it using JNI and track the Unix process.
Posts
