Utilizing a combination of several unique and advanced techniques, TDI has developed a novel mechanism for evaluating the trends and magnitude in major security areas such as malware, spam, and terrorism threat. The method is very generic and can be applied to almost any topic of interest. The mechanism developed by TDI is partially based on the ideas described in this research paper, which specifies how to use the concepts of compressibility and Kolmogorov complexity to develop a powerful data-mining mechanism that can extract information from the index of major search engines such as Google, Yahoo, MSN and others. This novel method can be used as an alternative to existing techniques for security threat and spam monitoring such as SANS Internet Threat Level, Symantec Internet Threat Meter and others. In addition, the TDI threat meter has the advantage that its predictions are based on the opinion of potentially billions of people around the world, captured in the corpus of major search engines. Thus, the results it produces are likely to be statistically significant.
The concepts of compressibility and Kolmogorov complexity can also be applied to the areas of intrusion detection, spam classification, and classification of unknown malware. The advantage is that this approach is not signature based and can detect malicious activity and traffic of unknown nature, image spam, as well as malware of unknown nature. Our tests indicate a high level of correct malware/spam classification with a very low level of false positives. TDI is in the process of developing modules that implement these methods and can easily be integrated into almost all enterprise products for intrusion detection (IDS/IPS), spam filtering, and malware protection.