Vortex cores in fluid mechanics are easy to visualize, yet difficult to detect numerically. Precise knowledge of these allows fluid dynamics researchers to study complex flow structures and allow for a better understanding of the turbulence transition process and the development and evolution of flow instabilities, to name but a few relevant areas. Various approaches such as the Q, delta, and swirling strength criterion have been proposed to visualize vortical flows, and these approaches can be used to detect vortex core locations. Using these methods can result in spuriously detected vortex cores and which can be balanced by a cutoff filter, making these methods lack robustness. To overcome this shortcoming, we propose a new approach using convolutional neural networks to detect flow structures directly from streamline plots, using the line integral convolution method. We show that our computer vision-based approach is able to reduce the number of false positives and negatives while removing the need for a cutoff. We validate our approach using the Taylor–Green vortex problem to generate input images for our network. We show that with an increasing number of images used for training, we are able to monotonically reduce the number of false positives and negatives. We then apply our trained network to a different flow problem where vortices are still reliably detected. Thus, our study presents a robust approach that allows for reliable vortex detection which is applicable to a wide range of flow scenarios.