It doesn’t seem that much of a stretch when you know that monks originated as hermit who tried to live ascetic lifestyles, often by retreating to caves far away from civilisation. This is just an evolution of that.