Big Data Processing Algorithms
We develop a series of programs that pull information from big data sources like the US Global Agricultural Trade System (GATS), parses it, normalizes it and loads it into a structured MySQL Database from where the CIH’s team generates multiple reports for decision makers.
KEY ACHIEVEMENTS
- Processing of millions of records with multithreading algorithms to achieve a high performance rate.
- Dealing with multiple sources of information: CSV files, JSON files, info pulled from the web and crawled organically.