Taiwan fish markets sell a wide variety of fish, and laypeople may have difficulty recognizing the fish species. The identification of fish species is still mostly based on illustrated handbooks, which is time-consuming when users lack experience. Automatic segmentation and recognition of fish images are important for the field of oceanography. However, in fish markets, the instability of light sources and changes in illumination influence the brightness and colors of fish. Moreover, fish markets often arrange fish together and cover them with ice to keep them fresh, thus increasing the difficulty of automatic fish recognition. This study presents a fish recognition system that combines a state-of-art instance segmentation method along with ResNet-based classification. An input image is first passed through the fish segmentation model, which crops the image into several images containing specific objects with a plain black background. Then the cropped images are assigned to a class by the fish classification model, which returns the predicted label of each image. A database of real fish images was collected from a fish market to verify the system. The experimental results revealed that the system achieved 85% Top-1 accuracy and 95% Top-5 accuracy on the test data set.