I found the same thing until I started strictly controlling the resources each container could consume, and also changing to a much beefier machine. Running a single project with a few images were fine, but more than that and the WSL connection would randomly crash or become unresponsive.

Databases in particular you need to watch: left unchecked they will absolutely hog RAM.