How can I optimize my Python codebase using multiprocessing or asyncio to speed up heavy text-processing tasks on our university high-performance computing cluster?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
How can I optimize my Python codebase using multiprocessing or asyncio to speed up heavy text-processing tasks on our university high-performance computing cluster?
1 real search · 1 real-search pass
Python multiprocessing ProcessPoolExecutor documentation chunksize shared memory asyncio CPU-bound Slurm HPC job arrays CPU binding best practicesGoogle ↗
OBSERVEDInstruction
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (1)
Classes come from the query-classifier (a local Laya model). 1 of 1 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT1 real searches · gpt-6
Python multiprocessing ProcessPoolExecutor documentation chunksize shared memory asyncio CPU-bound Slurm HPC job arrays CPU binding best practices Google ↗
- Parallel Python · docs.nersc.gov
- Python · docs.urcf.drexel.edu
- Parallel Python on a Single Node · rc.dartmouth.edu
- Python Performance Optimization - Python Best Practice Guide (WORK IN PROGRESS) · hpc.pages.naiss.se
- Best Practices · orcd-docs.mit.edu
- Multi-processing vs. Multi-threading · docs.hpc.msstate.edu
- multiprocessing — Process-based parallelism · docs.python.org
- Writing Performant NumPy Code with Multi-Core CPUs · numpy.org