Evaluate scaling based on MJCS load testing
Created by: dismantl
Since both the spider and scraper components make requests to the MJCS server, they are limited in their concurrency so as not to overload MJCS and make it unavailable to others. By default, both components are limited to 10 concurrent connections to MJCS. These defaults are arbitrary safe values; it would be better to determine safe concurrency levels based on load testing MJCS.
For each search of MJCS, the server reports the number of seconds it took to process the request:
If we were to periodically execute the same search on MJCS and track the number of seconds the search takes, and graph that versus the running concurrency of Case Harvester as we scale it up and down, that would help us see when increasing concurrency actually starts affecting MJCS server performance.