We study the differences between the processes of decoherence induced by chaotic and regular environments. For this we analyze a family of simple models that contain both regular and chaotic environments. In all cases the system of interest is a “quantum walker”, i.e., a quantum particle that can move on a lattice with a finite number of sites. The walker interacts with an environment which has a D-dimensional Hilbert space. The results we obtain suggest that regular and chaotic environments are not distinguishable from each other in a (short) time scale t^*, which scales with the dimensionality of the environment as t^*~log₂(D). However, chaotic environments continue to be effective over exponentially longer time scales while regular environments tend to reach saturation much sooner. We present both numerical and analytical results supporting this conclusion. The family of chaotic evolutions we consider includes the so-called quantum multibaker map as a particular case.