We present a generic data-driven method to address the problem of manipulating a three-dimensional (3-D) compliant object (CO) with heterogeneous physical properties in the presence of unknown disturbances. In this study, we do not assume a prior knowledge about the deformation behavior of the CO and type of the disturbance (e.g., internal or external). We also do not impose any constraints on the CO's physical properties (e.g., shape, mass, and stiffness). The proposed optimal iterative algorithm incorporates the provided visual feedback data to simultaneously learn and estimate the deformation behavior of the CO in order to accomplish the desired control objective. To demonstrate the capabilities and robustness of our algorithm, we fabricated two novel heterogeneous compliant phantoms and performed experiments on the da Vinci Research Kit. Experimental results demonstrated the adaptivity, robustness, and accuracy of the proposed method and, therefore, its suitability for a variety of medical and industrial applications involving CO manipulation.