corpus.blog
41554 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a087c8-0c26-720f-b589-4bf6b17280e5

Demystify RAM Usage in Multi-Process Data Loaders

ppwwyyxx.com · published 24 December 2022

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at ppwwyyxx.com Wayback capture
1blogs citing
First cited
24 Feb 2025
Most recent
24 Feb 2025
Rank this month
not ranked
Links from this post
32
Words captured
3,847

Cited by

Blog
In the post
Date
ericyuegu.com
Training AI to play Super Smash Bros. Melee Read at the source ↗
24 Feb 2025

Links from this post

Link
Host
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader
github.com
https://pytorch.org/docs/stable/data.html
pytorch.org
https://man7.org/linux/man-pages/man5/proc.5.html
man7.org
https://en.wikipedia.org/wiki/Unique_set_size
en.wikipedia.org
https://en.wikipedia.org/wiki/Resident_set_size
en.wikipedia.org
https://en.wikipedia.org/wiki/Proportional_set_size
en.wikipedia.org
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/common.py
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-naive.py
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-naive-log.txt
github.com
https://github.com/pytorch/pytorch/issues/13246
github.com
https://pytorch-dev-podcast.simplecast.com/episodes/dataloader-with-multiple-workers-leaks-memory
pytorch-dev-podcast.simplecast.com
https://en.wikipedia.org/wiki/Copy-on-write
en.wikipedia.org
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/795868a37446d61412b9a58dbb1b7c76e75d39c4/main-naive-log.txt
github.com
https://docs.python.org/3/library/gc.html
docs.python.org
https://github.com/facebookresearch/detectron2/commit/0cd0e7253f64fc40e9c2ee4617df6e4760d07974
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/795868a37446d61412b9a58dbb1b7c76e75d39c4/serialize.py
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-fork-numpyserialize-log.txt
github.com
https://bnikolic.co.uk/blog/python/parallelism/2019/11/13/python-forkserver-preload.html
bnikolic.co.uk
https://discuss.python.org/t/switching-default-multiprocessing-context-to-spawn-on-posix-as-well/
discuss.python.org
https://github.com/python/cpython/blob/3c033a2e6fbde56f904aeca138141be6564341cf/Lib/multiprocessing/reduction.py
github.com
https://github.com/pytorch/pytorch/blob/fd3a7264ae80332ba5ec8f60446e6d7a2c2276c1/torch/multiprocessing/reductions.py
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-torchserialize.py
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-spawn-torchserialize-log.txt
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/795868a37446d61412b9a58dbb1b7c76e75d39c4/main-spawn-torchserialize-log.txt
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-forkserver-torchserialize-log.txt
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-multigpu-naive.py
github.com
https://pytorch.org/docs/stable/elastic/run.html
pytorch.org
https://github.com/huggingface/accelerate
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/main-multigpu-sharedmem.py
github.com
https://github.com/facebookresearch/mobile-vision/pull/120
github.com
https://github.com/ppwwyyxx/RAM-multiprocess-dataloader/blob/main/serialize.py
github.com
https://peps.python.org/pep-0703/
peps.python.org