How to use Selenium and multithreads in Python3

Problem

We need to run several web tests using Selenium. If you ever use this framework before, you already know that only one task uses a lot of RAM and CPU resources (well, maybe we can blame Firefox or Chrome). In my case I needed to run over 2,000 different tests or URLs. Running the software in a i7 processor with 16 GB of RAM, it only took 30 processed to fill most of the available memory.

Solution

We need to find a way to run a fixed amount of tests without running out of memory or increasing the load of the CPU too much that would hang our system.

Let's use Python3:

Basically, what I do is read a line from a file, every line has two values, arg1 and arg2 separated by commas. Both are passed to the run function on line 17. This way you'll be reading 20 lines constantly.

When a worker is done another new one is added to the queue.

I hope this helps you understand how Processes/Queues work in Python.