First Public Sandbox Evaluation Using Venak Security’s AI Malware Simulator
Venak Security has conducted a public evaluation of major sandbox solutions using its AI Malware Simulator, marking an important step toward more structured and transparent testing of malware analysis environments.
Last year, Venak Security collaborated with three major cybersecurity companies to develop a sandbox evaluation framework designed to improve the way malicious software is analyzed and detected in controlled environments. The objective of this initiative was to establish a standardized methodology for assessing how sandbox technologies respond to emerging threats, including zero-day and AI-generated malware.
The resulting framework was developed with reference to industry best practices and guidance from the Anti-Malware Testing Standards Organization, known as AMTSO. By aligning the evaluation approach with recognized testing principles, the framework aims to support greater consistency, transparency, and reliability in the assessment of cybersecurity products.
This year, Venak Security applied the framework to evaluate several major publicly available sandbox solutions. The evaluation examined how these environments performed when exposed to simulated AI-generated malware behaviors under controlled and repeatable testing conditions.
This test is based on the Anti-Malware Testing Standards Organization (AMTSO) standard, AMTSO Test ID: AMTSO-LS1-TP204:
Venak Security introduced its AI Malware Simulator in September 2024, and since then, we have evaluated many cybersecurity products and solutions through private and public tests.
Sandbox technologies play a critical role in modern cybersecurity operations. They allow analysts, vendors, and security teams to observe suspicious files and behaviors before they affect production systems. However, comparing sandbox performance has often been difficult due to differences in testing methods, sample selection, scoring criteria, and reporting practices.
By introducing a structured public evaluation, Venak Security aims to contribute to a more consistent approach for measuring sandbox effectiveness. The use of an AI Malware Simulator also reflects the growing importance of artificial intelligence in both threat simulation and defensive security validation.
As AI-generated threats become more advanced and harder to detect, security tools must be evaluated against scenarios that better represent the evolving threat landscape.
This evaluation represents a meaningful milestone in the development of modern sandbox testing methodologies. It may also support future efforts to establish clearer standards for how sandbox solutions are assessed, compared, and improved across the cybersecurity industry.
Vendors Tested and Vendors That Opted Out
The initial scope of the evaluation covered the following vendors:
- OPSWAT MetaDefender
- VMRay TotalInsight
- Malwation THREAT.ZONE
- ANY.RUN Malware Sandbox
- ReversingLabs Spectra Analyze
- Check Point SandBlast
- Joe Sandbox Cloud
- CrowdStrike Falcon Sandbox
From the list above, Joe Sandbox, ReversingLabs, and VMRay opted out, and we had to remove Check Point due to technical difficulties. We hope these sandboxes companies will return for our second public test later in December.
Test Sample Creation Methodology
We have created following samples using our Pioneer AI Malware Simulator technology:
- AI-generated zero-day ransomware
- AI-generated infostealer malware
- Zero-day DLL sideloading exploits
- Packed and obfuscated AI-generated malware
- Evasive malware samples designed to bypass sandbox detection
All samples were validated prior to testing to ensure consistency and reproducibility.
Testing Methodology
Each test sample is uploaded to the sandbox environment being evaluated. The sandbox is then given approximately one to two minutes to complete its initial analysis and produce a detection verdict. Results are recorded based on the sandbox’s classification of the sample, including whether it is identified as malicious, suspicious, or benign during the initial analysis period.
How We Determine Pass or Fail
For malicious samples, a sandbox is considered to have passed if it assigns a risk score of 6/10, 51/100, or higher, or if it classifies the sample as High Risk, Malicious, or an equivalent high-confidence threat verdict.
For clean samples, a sandbox is considered to have passed if it assigns a risk score of 5/10, 50/100, or lower, or if it classifies the sample as Clean, Benign, No Threat Detected, or an equivalent non-malicious verdict.
These risk scores are defined by the vendors and reflect their assessment of whether a threat should be considered malicious or clean.
For the Sandbox Evasion Detection test, the sandbox is expected to identify the relevant MITRE ATT&CK technique associated with sandbox evasion activity. However, the sample itself should still be classified as Clean or No Threat Detected, since the objective of this test is to evaluate whether the sandbox can recognize evasion-related behavior without incorrectly classifying the sample as malicious.
If the sandbox displays “function here” in its analysis output during testing, it indicates that the sandbox successfully bypassed our evasion techniques.
Videos of All tests
At Venak Security, we believe consumers and vendors have the right to see the real footage of the tests we conduct. We do this mainly to ensure transparency in our testing process.
All vendors received these videos before they were made available to the public. Some of the errors seen in these videos have been fixed, and we have mentioned those in the test results section.
https://venaksecurity.com/wp-content/uploads/2026/07/Q2_2026_ANY.RUN_.zip
https://venaksecurity.com/wp-content/uploads/2026/07/Q2_2026_CrowdStrike.zip
https://venaksecurity.com/wp-content/uploads/2026/07/Q2_2026_OPSWAT.zip
https://venaksecurity.com/wp-content/uploads/2026/07/Q2_2026_Malwation.zip
Test Overview
AI-generated zero-day ransomware test
These samples were developed to evaluate sandbox detection and evasion capabilities. When executed, each sample attempts to determine whether it is running inside a sandboxed analysis environment. If sandbox-related indicators are detected, the sample immediately terminates execution.
If the sample continues running and performs its intended behavior, this indicates that the sandbox environment was not successfully identified by the sample. In this context, continued execution suggests that the sandbox’s anti-evasion mechanisms were effective against the detection methods used by that specific sample.
The samples include behaviors associated with the following MITRE ATT&CK references:
- 2.exe: S0129, S0396
- 4.exe: S0129, S0396, S0574, S0584
- 6.exe: S0129, S0396, S0574, S0584, S0595, S0606
- 8.exe: S0129, S0396, S0574, S0584, S0595, S0606, S0694, S1018
- 10.exe: S0129, S0396, S0574, S0584, S0595, S0606, S0694, S1018, S1130, T1562.004
- 11.exe: S0129, S0396, S0574, S0584, S0595, S0606, S0694, S1018, S1130, T1562.004, T1562.002
The samples were generated using Venak Security’s AI Malware Simulator and compiled as C++ executables.
Sample Video
Sandbox Evaluation Results
Test Summary
Our ransomware test is one of our flagship tests. These samples are very similar to those used in our previous AV/EDR test conducted in March 2026. However, we modified them using AI to behave differently in this test.
OPSWAT MetaDefender and Malwation THREAT.ZONE did a fantastic job detecting the samples. However, CrowdStrike and ANY.RUN were only able to detect 3 and 1 samples, respectively.
Regarding the ANY.RUN sandbox, our first few samples were able to detect the ANY.RUN sandbox environment and exited before executing. As a result, ANY.RUN was unable to run or analyze the executable. A spokesperson from ANY.RUN stated that the company is focusing on improving this area of the sandbox before we publish the results to the public.
CrowdStrike Falcon Sandbox performed better than ANY.RUN; however, it still failed to detect all of the samples.
One of the most important findings from this test relates to the AV/EDR tests we published in March 2026. At the time, some cybersecurity researchers questioned how CrowdStrike Falcon EDR could fail all of the ransomware tests. However, the results shown in our test video clearly show that CrowdStrike Falcon EDR classified all ransomware samples as clean. (Links available in the Videos of All Tests section above)
We hope these test videos resolve the controversy by providing clear evidence that the samples were classified as clean during our AV/EDR testing. These samples are not identical, but they belong to the same family and were specifically designed to operate in a sandbox environment while also running on a normal machine.
AI-generated infostealer malware
For these tests, we developed an infostealer designed to extract stored credentials from Chrome and Opera browsers. The Chrome credential extraction component was not developed by Venak Security; it is based on an open-source project authored by Alex Hagenah (https://github.com/xaitax). We enhanced this component using AI for testing purposes. The remaining samples used in the evaluation were generated by our internal AI Malware Simulator and compiled using Python.
Sample Video
Sandbox Evaluation Results
Test Summary
In this test, we used a variety of code samples written by our AI Malware Simulator that could potentially trigger alerts in sandbox environments. OPSWAT MetaDefender, CrowdStrike Falcon Sandbox, and Malwation THREAT.ZONE did a fantastic job detecting these samples.
However, regarding the ANY.RUN test results, we noticed an issue in the ANY.RUN sandbox that prevented samples from executing when they were compiled using a certain version of Python. This was very unusual to detect and almost impossible for a human tester to identify, which is exactly why our AI Malware Simulator developed samples using various programming languages.
We worked with the ANY.RUN team, and a spokesperson confirmed that the issue had been fixed. We retested the samples and verified that the error had been resolved. Currently, threats developed using all versions of Python can be executed and detected on ANY.RUN.
Zero-day DLL sideloading exploits
Attackers are increasingly using DLL sideloading techniques to execute malicious code by abusing trusted applications. By placing a rogue DLL alongside a legitimate executable, they can bypass security controls and gain stealthy execution on a system. This method is particularly effective because it blends in with normal software behavior, making detection more difficult. If you haven’t read our previous blog on this topic, we strongly recommend checking it out.
For this test, we have many options to choose from. Last year, we discovered more than 50+ software vendors that suffered from these vulnerabilities in their software. However, in order to publicly release them, it’s a bit challenging. This time, we used a vendor that has already been targeted by hackers many times before, and they have refused to fix their software in the past 4 months.
Sample Video
Sandbox Evaluation Results
Test Summary
In this test we were kind of expecting that most Sandboxes would fail. This type of attack is very new for all cybersecurity solutions (such as AVs, EDRs, AI SOC, Sandboxes, etc.) and this is an alarm for any vendors. We wanted to congratulate the Malwation team for what they’ve done and we were surprised by how they’ve done a great job detecting this test.
During our test we noticed that OPSWAT MetaDefender, CrowdStrike Falcon Sandbox and ANY.RUN Malware Sandbox had some issues running the sample. Spokespersons from OPSWAT and ANY.RUN reached out to us and mentioned that they were able to fix the issues. We have retested and confirmed that these issues were already fixed for these two platforms at the time of releasing this article.
Packed and obfuscated AI-generated malware
For this test, we developed a credit card data extraction malware simulator designed to identify sensitive information within a system, such as data stored in web browsers and password managers. Recently, many cybersecurity organizations have been targeted by similar attacks, making it essential to use such AI-generated tools to evaluate sandboxing capabilities effectively. After development, the executable was packed and obfuscated. All samples in this test include anti-VM, anti-sandbox, and anti-virtual machine techniques.
Sample Video
Sandbox Evaluation Results
Test Summary
In this test, we were expecting sandboxes that use emulation to perform better versus VM-based sandboxes, mainly because these threats do not do anything in the file system and are only active in memory. Both OPSWAT MetaDefender and CrowdStrike Falcon Sandbox have done fantastic jobs in these two tests.
Evasive malware samples designed to bypass sandbox detection
This is the most important part of our test, as if a sandbox is able to detect the malware using anti-sandbox mechanisms, it can effectively block or identify it.
We intentionally did not include any malicious activity in these samples; therefore, they are clean. However, we implemented logic that checks whether the sample is running inside a sandbox environment. We used five different techniques for this detection. This is why it is important to use our AI malware simulator to specifically test certain MITRE ATT&CK techniques. We can extend this approach to simulate more than 500 different MITRE ATT&CK methods.
The sandbox should report the sample as clean and display a message box such as “function here,” in order to pass our test.
Sample Video
Sandbox Evaluation Results
Test Summary
In this test, we expected all vendors to perform strongly, as detecting and analyzing evasive techniques is one of the primary purposes of running samples through sandboxes. All vendors did a fantastic job identifying our sandbox evasion techniques and demonstrated strong detection capabilities.
All Test Results
Sandbox Speed Test Results
How We Determine Speed
Sandbox speed is measured as the elapsed time between the defined start point (sample upload) and the stop point (sandbox report generation). To determine the overall sandbox speed, we calculate the average execution time across multiple samples and use this average as the representative speed measurement.
Speed Test Summary
Regarding the speed tests, OPSWAT was the fastest, with an average analysis time of approximately 10 seconds per sample, making it the best-performing solution in this evaluation. CrowdStrike was the slowest, averaging 2 to 3 minutes per sample. However, it still performed significantly better than Malwation and Any.Run in terms of detection during our tests.
Conclusion
We have heavily invested in our AI Malware Simulator, and we believe the future of cybersecurity depends on technologies like this. Detecting threats before cybercriminals do is becoming increasingly important. Cybercriminals already have access to uncensored AI models capable of generating malicious files at incredible speed. We strongly recommend that all security vendors invest heavily in AI-driven detection technologies to identify these threats more effectively. If you need assistance, Venak Security is here to help. Please check out our services here.
Thank you.
Leave a Reply