Flow-structure interactions of submerged or floating bodies can lead to undesired behavior in many marine and offshore engineering applications. In this paper, we consider a complex nonlinear dynamical system of unsteady wake flow interacting with a freely moving tugboat in open water. To meet the operational demands of compact, agile, and high power ship-handling, new hull forms of tugboats are designed with low length-to-beam ratios and rounded sterns. While the hydrodynamic design with low length-to-beam ratios provides improved directional controllability, it can be challenging for tractor tugboats due to the massively separated wake flow with the vortex shedding. These wake vortices can cause large fluctuating yaw moments and possibly a strong fluid-structure coupling via synchronization or lock-in. Of particular interest, the proposed study will focus on the physical mechanism and control of flow-induced oscillations with free-surface effects via our in-house fully-coupled three-dimensional fluid-structure-free-surface interaction solver. Stabilized finite element based methods will be employed to discretize the partial differential equations that arise from the mathematical modeling of the physical phenomena considered. We will begin with a fundamental understanding of coupled dynamics of a canonical geometry of a freely vibrating sphere at free-surface. The physical insight gained will then be applied to a realistic tugboat configuration. We aim to understand the fundamentals of vortex-shedding modes and the coupled dynamics pertaining to the flow-induced vibration (FIV) response of a freely vibrating sphere (a prototypical problem for a rounded tugboat) in all three spatial directions. To predict and analyze the vortex synchronization regimes and the wake patterns, the FIV response of the sphere at a low mass ratio is investigated over a broad range of reduced velocities and Reynolds numbers. We find that the sphere begins to move along a linear trajectory with hairpin vortex-shedding mode, eventually transforming into a circular trajectory with spiral mode in its stationary state for Re ∈ [2000–6000]. We systematically examine these mode transitions and the motion trajectories in the three degrees-of-freedom for higher Reynolds number up to 15,000 which has not been studied in detail in the literature. Finally, we will look into the effect of free surface on the FIV response of the sphere piercing the free surface and will link our fundamental results with a realistic configuration of tugboat undergoing vortex-induced oscillation with free surface effects.