Machine vision detection of foreign objects in coal using deep learning
-
摘要: 在煤炭生产加工过程中，由于开采条件的限制，原煤中混入了大量的异物，导致重介质系统管路的堵塞，严重影响了煤炭加工的生产效率，制约了煤炭质量的提高。为有效解决煤炭分选系统中异物对生产的影响，使用机器视觉技术完成异物检测。本文针对煤中异物的不规则形状和复杂特征，构建一种多尺度特征融合的语义分割网络，对实际生产现场采集的煤中异物图像进行端到端检测。网络中，编码器端采用残差卷积块提取异物图像特征和捕捉上下文信息，将池化操作得到的多尺度特征图谱在解码器端进行融合；将编码器与解码器使用跳层连接，更好地结合图像的背景语义信息，进行端到端的训练；通过类激活图可视化方法分析了模型误检原因，提出了一种损失函数用于缓解模型因煤矸石表面和背景干扰产生的误检情况，并使用条件随机场对网络分割结果进行细化，最终得到一幅二值图像。实验结果表明：该模型能够对煤矸系统中的异物进行有效分割，最终模型在测试集上的均交并比(MIOU)为77.83 %。Abstract: In the process of coal processing, due to the limitation of mining conditions, a large number of foreign objects are mixed into raw coal, which leads to clogging of heavy medium system pipelines, seriously affecting the production efficiency and restricting the improvement of coal quality. In order to effectively solve the impact of foreign objects in coal, machine vision technology is used to complete the detection and removal of foreign objects. Owing to irregular shapes and complex features for foreign objects in coal, a multi-scale feature fusion semantic segmentation network structure is proposed for end-to-end detection of images of foreign objects collected at the actual production site. In the network, the encoder uses residual convolutional blocks to extract features and captures context information, and the multi-scale feature map obtained by the pooling operation is fused at the decoder. The encoder and decoder are connected by layer jump to better combine the background semantic information of images for end-to-end training. The cause of model misdetection was analyzed through the visualization method of class activation map, a loss function was proposed to alleviate the model misdetection caused by the surface and background interference of coal and gangue, and the results were refined using conditional random fields to get a binary image. The experimental results show that the model can effectively segment the foreign bodies in the coal gangue system, and the mean intersection over union of the model on the test set is 77.83 %.
-
Key words:
- foreign objects detection /
- machine vision /
- deep learning /
- semantic segmentation /
- coal
-
表 1 主要设备清单
Table 1. List of major equipment
主要设备 型号 参数 相机 acA4096-40um/uc 分辨率4 096×2 168帧率42fps CPU i7 9600K 6核/3.70 GHz GPU TITAN XP 8G显存 
下载: 导出CSV
表 2 模型表现对比
Table 2. Comparison with other models
% 模型 正样本MIOU 总样本MIOU FCN-32 48.39 55.33 Segnet 51.08 68.11 Unet 52.45 71.62 Proposed Net 55.82 77.83
下载: 导出CSV
表 3 使用CRF前后数据对比
Table 3. Data Comparison with CRF
% 迭代次数 交叉熵损失函数 本文提出的损失函数 正样本MIOU 总样本MIOU 正样本MIOU 总样本MIOU 0 43.68 71.62 51.53 75.36 1 52.04 75.86 55.92 77.83 2 54.16 76.93 54.68 77.22 3 55.01 77.37 52.80 76.28 4 55.36 77.55 51.00 75.37 5 55.52 77.63 49.21 74.48
下载: 导出CSV
-
[1] 柴利文. 同煤集团降低出口煤杂物含量的措施[J]. 煤质技术, 2008(1): 4-5. doi: 10.3969/j.issn.1007-7677.2008.01.002Chai Liwen. Tongmei Group's measures to reduce the content of exported coal debris[J]. Coal Quality Technology, 2008(1): 4-5. doi: 10.3969/j.issn.1007-7677.2008.01.002 [2] 邓洪刚, 王晓薇. 多煤种混配条件下除杂物系统的应用与改进[J]. 中国新技术新产品, 2011(7): 32-32. doi: 10.3969/j.issn.1673-9957.2011.07.026Deng Honggang, Wang Xiaowei. Application and improvement of debris removal system under multi-coal blending conditions[J]. China New Technology & New Products, 2011(7): 32-32. doi: 10.3969/j.issn.1673-9957.2011.07.026 [3] 李文利, 朱子祺. 石圪台选煤厂杂物治理措施[J]. 煤炭加工与综合利用, 2011(4): 9-10. doi: 10.3969/j.issn.1005-8397.2011.04.003Li Wenli, Zhu Ziqi. Managing measures of sundry articles in getai coal preparation plant[J]. Coal Processing and Comprehensive Utilization, 2011(4): 9-10. doi: 10.3969/j.issn.1005-8397.2011.04.003 [4] 张万青, 李晓燕. 完善除杂设施严把产品质量关[J]. 煤质技术, 2009(2): 31-32. doi: 10.3969/j.issn.1007-7677.2009.02.012Zhang Wanqing, Li Xiaoyan. Discussion on advancing the eliminating impurities equipment and improving the quality of products[J]. Goal Quality and Technology, 2009(2): 31-32. doi: 10.3969/j.issn.1007-7677.2009.02.012 [5] 王卫东, 张康辉, 吕子奇, 等. 基于机器视觉的煤中杂物智能分选系统研究[J]. 选煤技术, 2020(2): 87-91. https://www.cnki.com.cn/Article/CJFDTOTAL-XMJS202002021.htmWang Weidong, Zhang Kanghui, Lü Ziqi, et al. Research on intelligent sorting system of coal debris based on machine vision[J]. Coal Preparation Technology, 2020(2): 87-91. https://www.cnki.com.cn/Article/CJFDTOTAL-XMJS202002021.htm [6] 栾永春. 基于视频分析的煤矿井下皮带异物检测系统[J]. 科技经济导刊, 2017(17): 70. https://www.cnki.com.cn/Article/CJFDTOTAL-JJKJ201717053.htmLuan Yongchun. Foreign body detection system of coal mine belt based on video analysis[J]. Science & Technology Economic Guide, 2017(17): 70. https://www.cnki.com.cn/Article/CJFDTOTAL-JJKJ201717053.htm [7] 郜振国. 煤矿井下运输异物检测关键技术研究[D]. 徐州: 中国矿业大学, 2018. [8] Rong D, Xie L, Ying Y, et al. Computer vision detection of foreign objects in walnuts using deep learning[J]. Computers and Electronics in Agriculture, 2019, 162: 1001-1010. doi: 10.1016/j.compag.2019.05.019 [9] Zhao X H, Guo X Y, Luo J, et al. Efficient detection method for foreign fibers in cotton[J]. Information Processing in Agriculture, 2018, 5(3): 320-328. doi: 10.1016/j.inpa.2018.04.002 [10] Ronneberger O, Fischer P, Brox T, et al. U-Net: convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer Assisted Intervention, 2015: 234-241. [11] Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2117-2125. [12] He K, Zhang X, Ren S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. doi: 10.1109/TPAMI.2015.2389824 [13] Krizhevsky A, Sutskever I, Hinton G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems, 2012: 1097-1105. [14] Long J, Shelhamer E, Darrell T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431-3440. [15] Badrinarayanan V, Kendall A, Cipolla R.Segnet: A deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495. doi: 10.1109/TPAMI.2016.2644615 [16] Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848. doi: 10.1109/TPAMI.2017.2699184 [17] 张帆, 李亚杰, 孙晓辉. 无线感知与视觉融合的井下目标跟踪定位方法[J]. 矿业科学学报, 2018, 3(5): 484-491. http://www.cnki.com.cn/Article/CJFDTotal-KYKX201805009.htmZhang Fan, Li Yajie, Sun Xiaohui. A novel method of mine target tracking and location based on wireless sensor and visual recognition[J]. Journal of Mining Science and Technology, 2018, 3(5): 484-491. http://www.cnki.com.cn/Article/CJFDTotal-KYKX201805009.htm [18] 郑晶, 曹子原, 姜天琪, 等. 基于深度信念神经网络的微震波到时拾取方法[J]. 矿业科学学报, 2018, 3(6): 521-526. http://www.cnki.com.cn/Article/CJFDTotal-KYKX201806002.htmZheng Jing, Cao Ziyuan, Jiang Tianqi, et al. Deep belief neural network-based arrival picking for micro- seismic data[J]. Journal of Mining Science and Technology, 2018, 3(6): 521-526. http://www.cnki.com.cn/Article/CJFDTotal-KYKX201806002.htm [19] 杨小彬, 王逍遥, 周世禄, 等. 基于改进广义回归神经网络的工作面低氧预测模型研究[J]. 矿业科学学报, 2019, 4(5): 434-440. doi: 10.19606/j.cnki.jmst.2019.05.008Yang Xiaobin, Wang Xiaoyao, Zhou Shilu, et al. Prediction model of working face hypoxia based on improved generalized regression neural network[J]. Journal of Mining Science and Technology, 2019, 4(5): 434-440. doi: 10.19606/j.cnki.jmst.2019.05.008 [20] Guan S, Khan A A, Sikdar S, et al. Fully Dense UNet for 2-D sparse photoacoustic tomography artifact removal[J]. IEEE Journal of Biomedical and Health Informatics, 2019, 24(2): 568-576. http://www.ncbi.nlm.nih.gov/pubmed/31021809 [21] 吴守鹏. 基于机器视觉的运煤皮带异物识别方法研究[D]. 徐州: 中国矿业大学, 2019. [22] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778. [23] 左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6): 718-726. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906007.htmZuo Zongcheng, Zhang Wen, Zhang Dongying. A remote sensing image semantic segmentation method by combining deformable convolution with conditional random fields[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 718-726. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906007.htm [24] 张新峰, 郭宇桐, 蔡轶珩, 等. 基于DCNN和全连接CRF的舌图像分割算法[J]. 北京航空航天大学学报, 2019, 45(12): 2364-2374. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK201912003.htmZhang Xinfeng, Guo Yutong, Cai Yiheng, et al. A tongue image segmentation algorithm based on DCNN and fully connected CRF[J]. Journal of Beijing University of Aeronautics and Astronautics, 2019, 45(12): 2364-2374. https://www.cnki.com.cn/Article/CJFDTOTAL-BJHK201912003.htm [25] 李承珊. 基于深度卷积神经网络的图像语义分割研究[D]. 北京: 中国科学院光电技术研究所, 2019. -
点击查看大图
图(11) / 表(3)
计量
- 文章访问数: 1147
- HTML全文浏览量: 281
- PDF下载量: 88
- 被引次数: 0