How to configure Neoload Load generation Agents to support workload?

I am getting issues with the load generator agents failing to start and tests aborting at the start.
I can run small number of users ok. I first tried to run on agent on same platform as Controller but hit issues.
Configured a remote agent but as I'm increasing workload am getting the same issue.
I have a datapool that seems ok at 10 M, needs remote at 20 M and at 100 M definitely needs 2 agents. The number of Vusers is only 20.
The config for the agents is straight out of the box. Am on Windows Server 2012, cpu is about 3% and have 124GB of RAM. Using Neoload 6.6.