We need to run several web tests using Selenium. If you ever use this framework before, you already know that only one task uses a lot of RAM and CPU resources (well, maybe we can blame Firefox or Chrome). In my case I needed to run over 2,000 different tests or URLs. Running the software in a i7 processor with 16 GB of RAM, it only took 30 processed to fill most of the available memory.
We need to find a way to run a fixed amount of tests without running out of memory or increasing the load of the CPU too much that would hang our system.
Let's use Python3:
Basically, what I do is read a line from a file, every line has two values, arg1 and arg2 separated by commas. Both are passed to the run function on line 17. This way you'll be reading 20 lines constantly.
When a worker is done another new one is added to the queue.
I hope this helps you understand how Processes/Queues work in Python.