Abstract: In the process of coal processing, due to the limitation of mining conditions, a large number of foreign objects are mixed into raw coal, which leads to clogging of heavy medium system pipelines, seriously affecting the production efficiency and restricting the improvement of coal quality. In order to effectively solve the impact of foreign objects in coal, machine vision technology is used to complete the detection and removal of foreign objects. Owing to irregular shapes and complex features for foreign objects in coal, a multi-scale feature fusion semantic segmentation network structure is proposed for end-to-end detection of images of foreign objects collected at the actual production site. In the network, the encoder uses residual convolutional blocks to extract features and captures context information, and the multi-scale feature map obtained by the pooling operation is fused at the decoder. The encoder and decoder are connected by layer jump to better combine the background semantic information of images for end-to-end training. The cause of model misdetection was analyzed through the visualization method of class activation map, a loss function was proposed to alleviate the model misdetection caused by the surface and background interference of coal and gangue, and the results were refined using conditional random fields to get a binary image. The experimental results show that the model can effectively segment the foreign bodies in the coal gangue system, and the mean intersection over union of the model on the test set is 77.83 %.