The combination of polymers with nano- or microsized solid materials displays novel and often enhanced properties compared to the traditional materials. They can open up possibilities for new technological applications. Materials whose physical properties can be varied by application of magnetic fields belong to a specific class of smart materials. The broad family of magnetic field-controllable soft materials includes ferrofluids, magneto-rheological fluids, magnetic gels, and magnetic elastomers. The magnetic gels and elastomers (magnetoelasts) represent a new type of composite and consist of small magnetic particles, usually in the nanometer to micron range, dispersed in a highly elastic polymeric matrix. The magnetic particles can be incorporated into the elastic body either randomly or in ordered structure. If a uniform magnetic field is applied to the reactive mixture during the cross-linking process, particle chains form and become locked into the elastomer. The resulting composites exhibit anisotropic properties. Combination of magnetic and elastic properties leads to a number of striking phenomena that are exhibited in response to impressed magnetic fields. The magnetic particles couple the shape and the elastic modulus with the external magnetic field. Giant deformational effects, high elasticity, anisotropic elastic and swelling properties, and quick response to magnetic fields open new opportunities for using such materials for various applications. Since the magnetic fields are convenient stimuli from the point of signal control, the magnetoelasts are promising smart materials in engineering due to their real-time controllable elastic properties. More recently, increasing interest has been devoted to exploration of multiresponsive magnetic polymers, which exhibit sensitivity to several external stimuli. Micro- and nanospheres that combine both magnetic, temperature, and pH sensitivity were also elaborated and studied. These new results provide novel possibilities for preparation of more complex magnetic field-responsive materials like membranes with on/off switching control. In this article, we review recent advances in mechanical and swelling behavior of magnetic field-responsive soft materials, including flexible polymer networks and gels.