| 网站首页 | 业界新闻 | 小组 | 威客 | 人才 | 下载频道 | 博客 | 代码贴 | 在线编程 | 编程论坛
欢迎加入我们,一同切磋技术
用户名:   
 
密 码:  
共有 979 人关注过本帖
标题:急,进程池pool apply_async 子进程无法启动
只看楼主 加入收藏
cbd666
Rank: 1
等 级:新手上路
帖 子:29
专家分:0
注 册:2019-9-26
结帖率:66.67%
收藏
 问题点数:0 回复次数:1 
急,进程池pool apply_async 子进程无法启动
自己弄了一天多了,还是无法解决,头疼的要死

Task类
程序代码:
class Task(models.Model):
    statusItemChoices = (
        (0, '等待中'),
        (1, '运行中'),
        (2, '完成')
    )
    matchMethodChoices = (
        (0, '模糊匹配'),
        (1, '精确匹配'),
        (2, '单词匹配')
    )

    name = models.CharField(
        max_length=50, null=False, blank=False, verbose_name=u'任务名'
    )
    keywords = models.TextField(null=False, blank=False, verbose_name='关键词')
    match_method = models.IntegerField(choices=matchMethodChoices, default=0)
    pages = models.IntegerField(default=5, null=False, verbose_name='爬取页数')
    interval = models.IntegerField(default=60, null=False, verbose_name='爬取间隔(分钟)')
    ignore_org = models.TextField(null=True, default='', verbose_name='忽略指定组织或账号下的代码')
    ignore_repo = models.TextField(null=True, default='', verbose_name='忽略某类仓库下的代码, 如 ')
    status = models.IntegerField(choices=statusItemChoices, default=0)
    start_time = models.DateTimeField(null=True)
    finished_time = models.DateTimeField(null=True)
    mail = models.TextField(null=True, default='', verbose_name='通知邮箱列表')


最一开始,随着任务量的增多,现在一次性每次都要开启200+个Task进程,原来的代码如下

程序代码:
import logging
import time
from django.core.management.base import BaseCommand
from django.db import close_old_connections
from django.db.models import Q
from django.conf import settings
import sys
from multiprocessing import Process,Pool
from github_monitor.apps.monitor.models.token import Token
from github_monitor.apps.monitor.models.task import Task
from github_monitor.apps.monitor.processors import TaskProcessor
if sys.version >= '3.8':
    import multiprocessing as mp
    mp.set_start_method('fork')

logger = logging.getLogger(__name__)
class Command(BaseCommand):

    task_dict = dict()
    INTERVAL = 60
    RS = settings.RS

    def handle(self, *args, **options):

        def _process(_processor):
            _processor.process()
        while True:
            close_old_connections()
            for token in Token.objects.all():
                key = 'token:%s' % token.value
                if not self.RS.exists(key):
                    self.RS.hset(key, 'reset', '')

            #原版
            for task in Task.objects.filter(~Q(id__in=self.task_dict)):  # 第一次遍历所有的任务
                 processor = TaskProcessor(task)
                 p = Process(target=_process, args=(processor,))
                 p.start()
                 self.task_dict[task.id] = p

            # 对于已删除的任务, 将其进程终止掉
            task_dict_copy = self.task_dict.copy()
            for task_id in task_dict_copy:
                 close_old_connections()
                 if not Task.objects.filter(id=task_id).count():
                     # task be deleted
                     print('terminate')
                     self.task_dict[task_id].terminate()
                     self.task_dict.pop(task_id)
            time.sleep(self.INTERVAL)


这里的TaskProcessor类的启动函数

程序代码:
    # 主程序入口
    def process(self):
        while True:

            connection.close()
            self.email_results = []
            self.task.refresh_from_db()
            self.task.status = 1
            self.task.start_time = timezone.now()
            self.task.finished_time = None
            self.task.save()
            keyword_list = self.task.keywords.split('\n')
            for keyword in keyword_list:
                _thread = Thread(target=self.search_by_keyword_thread, args=(keyword, ))
                _thread.start()
                self.thread_pool.append(_thread)
            for th in self.thread_pool:
                th.join()
            connection.close()
            self.task.status = 2
            self.task.finished_time = timezone.now()
            self.task.save()
            try:
                self.send_email()
            except Exception as e:
                logger.exception(e)
                pass

            time.sleep(self.task.interval * 60)


目前的情况就是每次运行总是只有100+个启动,剩下的就一直没有动静,于是我这里打印了每个启动Process的状态,如下所示

===============
{1: <Process(Process-270, started)>, 2: <Process(Process-269, started)>, 3: <Process(Process-268, started)>, 4: <Process(Process-267, started)>, 5: <Process(Process-266, started)>, 6: <Process(Process-265, started)>, 7: <Process(Process-264, stopped[1])>, 8: <Process(Process-263, stopped[1])>, 9: <Process(Process-262, stopped[1])>, 10: <Process(Process-261, stopped[1])>, 11: <Process(Process-260, stopped[1])>, 12: <Process(Process-259, stopped[1])>, 13: <Process(Process-258, stopped[1])>, 14: <Process(Process-257, stopped[1])>, 15: <Process(Process-256, stopped[1])>, 16: <Process(Process-255, stopped[1])>, 17: <Process(Process-254, stopped[1])>, 18: <Process(Process-253, stopped[1])>, 19: <Process(Process-252, stopped[1])>, 20: <Process(Process-251, stopped[1])>, 21: <Process(Process-250, stopped[1])>, 22: <Process(Process-249, stopped[1])>, 23: <Process(Process-248, stopped[1])>, 24: <Process(Process-247, stopped[1])>, 25: <Process(Process-246, stopped[1])>, 26: <Process(Process-245, stopped[1])>, 27: <Process(Process-244, stopped[1])>, 28: <Process(Process-243, stopped[1])>, 29: <Process(Process-242, stopped[1])>, 30: <Process(Process-241, stopped[1])>, 31: <Process(Process-240, stopped[1])>, 32: <Process(Process-239, stopped[1])>, 33: <Process(Process-238, stopped[1])>, 34: <Process(Process-237, stopped[1])>, 35: <Process(Process-236, stopped[1])>, 36: <Process(Process-235, stopped[1])>, 37: <Process(Process-234, stopped[1])>, 38: <Process(Process-233, stopped[1])>, 39: <Process(Process-232, stopped[1])>, 40: <Process(Process-231, stopped[1])>, 41: <Process(Process-230, stopped[1])>, 42: <Process(Process-229, stopped[1])>, 43: <Process(Process-228, stopped[1])>, 44: <Process(Process-227, stopped[1])>, 45: <Process(Process-226, stopped[1])>, 46: <Process(Process-225, stopped[1])>, 47: <Process(Process-224, stopped[1])>, 48: <Process(Process-223, stopped[1])>, 49: <Process(Process-222, stopped[1])>, 50: <Process(Process-221, stopped[1])>, 51: <Process(Process-220, stopped[1])>, 52: <Process(Process-219, stopped[1])>, 53: <Process(Process-218, stopped[1])>, 54: <Process(Process-217, stopped[1])>, 55: <Process(Process-216, stopped[1])>, 56: <Process(Process-215, stopped[1])>, 57: <Process(Process-214, stopped[1])>, 58: <Process(Process-213, stopped[1])>, 59: <Process(Process-212, stopped[1])>, 60: <Process(Process-211, stopped[1])>, 61: <Process(Process-210, stopped[1])>, 62: <Process(Process-209, stopped[1])>, 63: <Process(Process-208, stopped[1])>, 64: <Process(Process-207, stopped[1])>, 65: <Process(Process-206, stopped[1])>, 66: <Process(Process-205, stopped[1])>, 67: <Process(Process-204, stopped[1])>, 68: <Process(Process-203, stopped[1])>, 69: <Process(Process-202, stopped[1])>, 70: <Process(Process-201, stopped[1])>, 71: <Process(Process-200, stopped[1])>, 72: <Process(Process-199, stopped[1])>, 73: <Process(Process-198, stopped[1])>, 74: <Process(Process-197, stopped[1])>, 75: <Process(Process-196, stopped[1])>, 76: <Process(Process-195, stopped[1])>, 77: <Process(Process-194, stopped[1])>, 78: <Process(Process-193, stopped[1])>, 79: <Process(Process-192, stopped[1])>, 81: <Process(Process-191, stopped[1])>, 82: <Process(Process-190, stopped[1])>, 83: <Process(Process-189, stopped[1])>, 84: <Process(Process-188, stopped[1])>, 85: <Process(Process-187, stopped[1])>, 86: <Process(Process-186, stopped[1])>, 87: <Process(Process-185, stopped[1])>, 88: <Process(Process-184, stopped[1])>, 89: <Process(Process-183, stopped[1])>, 90: <Process(Process-182, stopped[1])>, 91: <Process(Process-181, stopped[1])>, 92: <Process(Process-180, stopped[1])>, 93: <Process(Process-179, stopped[1])>, 94: <Process(Process-178, stopped[1])>, 95: <Process(Process-177, stopped[1])>, 96: <Process(Process-176, stopped[1])>, 97: <Process(Process-175, stopped[1])>, 98: <Process(Process-174, stopped[1])>, 99: <Process(Process-173, stopped[1])>, 100: <Process(Process-172, stopped[1])>, 101: <Process(Process-171, stopped[1])>, 102: <Process(Process-170, stopped[1])>, 103: <Process(Process-169, stopped[1])>, 104: <Process(Process-168, stopped[1])>, 105: <Process(Process-167, stopped[1])>, 106: <Process(Process-166, stopped[1])>, 107: <Process(Process-165, stopped[1])>, 108: <Process(Process-164, stopped[1])>, 109: <Process(Process-163, stopped[1])>, 110: <Process(Process-162, stopped[1])>, 111: <Process(Process-161, stopped[1])>, 112: <Process(Process-160, stopped[1])>, 113: <Process(Process-159, stopped[1])>, 114: <Process(Process-158, stopped[1])>, 115: <Process(Process-157, stopped[1])>, 116: <Process(Process-156, stopped[1])>, 117: <Process(Process-155, stopped[1])>, 118: <Process(Process-154, stopped[1])>, 119: <Process(Process-153, stopped[1])>, 120: <Process(Process-152, stopped[1])>, 121: <Process(Process-151, stopped[1])>, 122: <Process(Process-150, stopped[1])>, 123: <Process(Process-149, stopped[1])>, 124: <Process(Process-148, stopped[1])>, 125: <Process(Process-147, stopped[1])>, 126: <Process(Process-146, stopped[1])>, 127: <Process(Process-145, stopped[1])>, 128: <Process(Process-144, stopped[1])>, 129: <Process(Process-143, stopped[1])>, 130: <Process(Process-142, stopped[1])>, 131: <Process(Process-141, stopped[1])>, 132: <Process(Process-140, stopped[1])>, 133: <Process(Process-139, stopped[1])>, 134: <Process(Process-138, stopped[1])>, 135: <Process(Process-137, stopped[1])>, 136: <Process(Process-136, stopped[1])>, 137: <Process(Process-135, stopped[1])>, 138: <Process(Process-134, stopped[1])>, 139: <Process(Process-133, stopped[1])>, 140: <Process(Process-132, stopped[1])>, 141: <Process(Process-131, stopped[1])>, 142: <Process(Process-130, stopped[1])>, 143: <Process(Process-129, stopped[1])>, 144: <Process(Process-128, stopped[1])>, 145: <Process(Process-127, stopped[1])>, 146: <Process(Process-126, stopped[1])>, 147: <Process(Process-125, stopped[1])>, 148: <Process(Process-124, stopped[1])>, 149: <Process(Process-123, stopped[1])>, 150: <Process(Process-122, stopped[1])>, 151: <Process(Process-121, stopped[1])>, 152: <Process(Process-120, stopped[1])>, 153: <Process(Process-119, stopped[1])>, 154: <Process(Process-118, stopped[1])>, 155: <Process(Process-117, stopped[1])>, 156: <Process(Process-116, stopped[1])>, 157: <Process(Process-115, stopped[1])>, 158: <Process(Process-114, stopped[1])>, 159: <Process(Process-113, stopped[1])>, 160: <Process(Process-112, stopped[1])>, 161: <Process(Process-111, stopped[1])>, 162: <Process(Process-110, stopped[1])>, 163: <Process(Process-109, started)>, 164: <Process(Process-108, started)>, 165: <Process(Process-107, stopped[1])>, 166: <Process(Process-106, stopped[1])>, 167: <Process(Process-105, stopped[1])>, 168: <Process(Process-104, started)>, 169: <Process(Process-103, stopped[1])>, 170: <Process(Process-102, started)>, 171: <Process(Process-101, started)>, 172: <Process(Process-100, started)>, 173: <Process(Process-99, started)>, 174: <Process(Process-98, started)>, 175: <Process(Process-97, started)>, 176: <Process(Process-96, started)>, 177: <Process(Process-95, started)>, 178: <Process(Process-94, started)>, 179: <Process(Process-93, started)>, 180: <Process(Process-92, started)>, 181: <Process(Process-91, started)>, 182: <Process(Process-90, started)>, 183: <Process(Process-89, started)>, 184: <Process(Process-88, started)>, 185: <Process(Process-87, started)>, 186: <Process(Process-86, started)>, 187: <Process(Process-85, started)>, 188: <Process(Process-84, started)>, 189: <Process(Process-83, started)>, 190: <Process(Process-82, started)>, 191: <Process(Process-81, started)>, 192: <Process(Process-80, started)>, 193: <Process(Process-79, started)>, 194: <Process(Process-78, started)>, 195: <Process(Process-77, started)>, 196: <Process(Process-76, started)>, 197: <Process(Process-75, started)>, 198: <Process(Process-74, started)>, 199: <Process(Process-73, started)>, 200: <Process(Process-72, started)>, 201: <Process(Process-71, started)>, 202: <Process(Process-70, started)>, 203: <Process(Process-69, started)>, 204: <Process(Process-68, started)>, 205: <Process(Process-67, started)>, 206: <Process(Process-66, started)>, 207: <Process(Process-65, started)>, 208: <Process(Process-64, started)>, 209: <Process(Process-63, started)>, 210: <Process(Process-62, started)>, 211: <Process(Process-61, started)>, 212: <Process(Process-60, started)>, 213: <Process(Process-59, started)>, 214: <Process(Process-58, started)>, 215: <Process(Process-57, started)>, 216: <Process(Process-56, started)>, 217: <Process(Process-55, started)>, 218: <Process(Process-54, started)>, 219: <Process(Process-53, started)>, 220: <Process(Process-52, started)>, 221: <Process(Process-51, started)>, 222: <Process(Process-50, started)>, 223: <Process(Process-49, started)>, 224: <Process(Process-48, started)>, 225: <Process(Process-47, started)>, 226: <Process(Process-46, started)>, 227: <Process(Process-45, started)>, 228: <Process(Process-44, started)>, 229: <Process(Process-43, started)>, 230: <Process(Process-42, started)>, 231: <Process(Process-41, started)>, 232: <Process(Process-40, started)>, 233: <Process(Process-39, started)>, 234: <Process(Process-38, started)>, 235: <Process(Process-37, started)>, 236: <Process(Process-36, started)>, 237: <Process(Process-35, started)>, 238: <Process(Process-34, started)>, 239: <Process(Process-33, started)>, 240: <Process(Process-32, started)>, 241: <Process(Process-31, started)>, 242: <Process(Process-30, started)>, 243: <Process(Process-29, started)>, 244: <Process(Process-28, started)>, 245: <Process(Process-27, started)>, 246: <Process(Process-26, started)>, 247: <Process(Process-25, started)>, 248: <Process(Process-24, started)>, 249: <Process(Process-23, started)>, 250: <Process(Process-22, started)>, 251: <Process(Process-21, started)>, 252: <Process(Process-20, started)>, 253: <Process(Process-19, started)>, 254: <Process(Process-18, started)>, 255: <Process(Process-17, started)>, 256: <Process(Process-16, started)>, 257: <Process(Process-15, started)>, 258: <Process(Process-14, started)>, 259: <Process(Process-13, started)>, 260: <Process(Process-12, started)>, 261: <Process(Process-11, started)>, 262: <Process(Process-10, started)>, 263: <Process(Process-9, started)>, 264: <Process(Process-8, started)>, 265: <Process(Process-7, started)>, 266: <Process(Process-6, started)>, 267: <Process(Process-5, started)>, 268: <Process(Process-4, started)>, 269: <Process(Process-3, started)>, 270: <Process(Process-2, started)>, 272: <Process(Process-1, started)>}===============


接着我就遍历了存活状态

程序代码:
<Process(Process-270, started)>True
<Process(Process-269, started)>True
<Process(Process-268, started)>True
<Process(Process-267, started)>True
<Process(Process-266, started)>True
<Process(Process-265, stopped[1])>False
<Process(Process-264, stopped[1])>False
<Process(Process-263, stopped[1])>False
<Process(Process-262, stopped[1])>False
<Process(Process-261, stopped[1])>False
<Process(Process-260, stopped[1])>False
<Process(Process-259, stopped[1])>False
<Process(Process-258, stopped[1])>False
<Process(Process-257, stopped[1])>False
<Process(Process-256, stopped[1])>False
<Process(Process-255, stopped[1])>False
<Process(Process-254, stopped[1])>False
<Process(Process-253, stopped[1])>False
<Process(Process-252, stopped[1])>False
<Process(Process-251, stopped[1])>False
<Process(Process-250, stopped[1])>False
<Process(Process-249, stopped[1])>False
<Process(Process-248, stopped[1])>False
<Process(Process-247, stopped[1])>False
<Process(Process-246, stopped[1])>False
<Process(Process-245, stopped[1])>False
<Process(Process-244, stopped[1])>False
<Process(Process-243, stopped[1])>False
<Process(Process-242, stopped[1])>False
<Process(Process-241, stopped[1])>False
<Process(Process-240, stopped[1])>False
<Process(Process-239, stopped[1])>False
<Process(Process-238, stopped[1])>False
<Process(Process-237, stopped[1])>False
<Process(Process-236, stopped[1])>False
<Process(Process-235, stopped[1])>False
<Process(Process-234, stopped[1])>False
<Process(Process-233, stopped[1])>False
<Process(Process-232, stopped[1])>False
<Process(Process-231, stopped[1])>False
<Process(Process-230, stopped[1])>False
<Process(Process-229, stopped[1])>False
<Process(Process-228, stopped[1])>False
<Process(Process-227, stopped[1])>False
<Process(Process-226, stopped[1])>False
<Process(Process-225, stopped[1])>False
<Process(Process-224, stopped[1])>False
<Process(Process-223, stopped[1])>False
<Process(Process-222, stopped[1])>False
<Process(Process-221, stopped[1])>False
<Process(Process-220, stopped[1])>False
<Process(Process-219, stopped[1])>False
<Process(Process-218, stopped[1])>False
<Process(Process-217, stopped[1])>False
<Process(Process-216, stopped[1])>False
<Process(Process-215, stopped[1])>False
<Process(Process-214, stopped[1])>False
<Process(Process-213, stopped[1])>False
<Process(Process-212, stopped[1])>False
<Process(Process-211, stopped[1])>False
<Process(Process-210, stopped[1])>False
<Process(Process-209, stopped[1])>False
<Process(Process-208, stopped[1])>False
<Process(Process-207, stopped[1])>False
<Process(Process-206, stopped[1])>False
<Process(Process-205, stopped[1])>False
<Process(Process-204, stopped[1])>False
<Process(Process-203, stopped[1])>False
<Process(Process-202, stopped[1])>False
<Process(Process-201, stopped[1])>False
<Process(Process-200, stopped[1])>False
<Process(Process-199, stopped[1])>False
<Process(Process-198, stopped[1])>False
<Process(Process-197, stopped[1])>False
<Process(Process-196, stopped[1])>False
<Process(Process-195, stopped[1])>False
<Process(Process-194, stopped[1])>False
<Process(Process-193, stopped[1])>False
<Process(Process-192, stopped[1])>False
<Process(Process-191, stopped[1])>False
<Process(Process-190, stopped[1])>False
<Process(Process-189, stopped[1])>False
<Process(Process-188, stopped[1])>False
<Process(Process-187, stopped[1])>False
<Process(Process-186, stopped[1])>False
<Process(Process-185, stopped[1])>False
<Process(Process-184, stopped[1])>False
<Process(Process-183, stopped[1])>False
<Process(Process-182, stopped[1])>False
<Process(Process-181, stopped[1])>False
<Process(Process-180, stopped[1])>False
<Process(Process-179, stopped[1])>False
<Process(Process-178, stopped[1])>False
<Process(Process-177, stopped[1])>False
<Process(Process-176, stopped[1])>False
<Process(Process-175, stopped[1])>False
<Process(Process-174, stopped[1])>False
<Process(Process-173, stopped[1])>False
<Process(Process-172, stopped[1])>False
<Process(Process-171, stopped[1])>False
<Process(Process-170, stopped[1])>False
<Process(Process-169, stopped[1])>False
<Process(Process-168, stopped[1])>False
<Process(Process-167, stopped[1])>False
<Process(Process-166, stopped[1])>False
<Process(Process-165, stopped[1])>False
<Process(Process-164, stopped[1])>False
<Process(Process-163, stopped[1])>False
<Process(Process-162, stopped[1])>False
<Process(Process-161, stopped[1])>False
<Process(Process-160, stopped[1])>False
<Process(Process-159, stopped[1])>False
<Process(Process-158, stopped[1])>False
<Process(Process-157, stopped[1])>False
<Process(Process-156, stopped[1])>False
<Process(Process-155, stopped[1])>False
<Process(Process-154, stopped[1])>False
<Process(Process-153, stopped[1])>False
<Process(Process-152, stopped[1])>False
<Process(Process-151, stopped[1])>False
<Process(Process-150, stopped[1])>False
<Process(Process-149, stopped[1])>False
<Process(Process-148, stopped[1])>False
<Process(Process-147, stopped[1])>False
<Process(Process-146, stopped[1])>False
<Process(Process-145, stopped[1])>False
<Process(Process-144, stopped[1])>False
<Process(Process-143, stopped[1])>False
<Process(Process-142, stopped[1])>False
<Process(Process-141, stopped[1])>False
<Process(Process-140, stopped[1])>False
<Process(Process-139, stopped[1])>False
<Process(Process-138, stopped[1])>False
<Process(Process-137, stopped[1])>False
<Process(Process-136, stopped[1])>False
<Process(Process-135, stopped[1])>False
<Process(Process-134, stopped[1])>False
<Process(Process-133, stopped[1])>False
<Process(Process-132, stopped[1])>False
<Process(Process-131, stopped[1])>False
<Process(Process-130, stopped[1])>False
<Process(Process-129, stopped[1])>False
<Process(Process-128, stopped[1])>False
<Process(Process-127, stopped[1])>False
<Process(Process-126, stopped[1])>False
<Process(Process-125, stopped[1])>False
<Process(Process-124, stopped[1])>False
<Process(Process-123, stopped[1])>False
<Process(Process-122, stopped[1])>False
<Process(Process-121, stopped[1])>False
<Process(Process-120, stopped[1])>False
<Process(Process-119, stopped[1])>False
<Process(Process-118, stopped[1])>False
<Process(Process-117, stopped[1])>False
<Process(Process-116, stopped[1])>False
<Process(Process-115, stopped[1])>False
<Process(Process-114, stopped[1])>False
<Process(Process-113, stopped[1])>False
<Process(Process-112, stopped[1])>False
<Process(Process-111, stopped[1])>False
<Process(Process-110, stopped[1])>False
<Process(Process-109, started)>True
<Process(Process-108, started)>True
<Process(Process-107, stopped[1])>False
<Process(Process-106, stopped[1])>False
<Process(Process-105, stopped[1])>False
<Process(Process-104, started)>True
<Process(Process-103, stopped[1])>False
<Process(Process-102, started)>True
<Process(Process-101, started)>True
<Process(Process-100, started)>True
<Process(Process-99, started)>True
<Process(Process-98, started)>True
<Process(Process-97, started)>True
<Process(Process-96, started)>True
<Process(Process-95, started)>True
<Process(Process-94, started)>True
<Process(Process-93, started)>True
<Process(Process-92, started)>True
<Process(Process-91, started)>True
<Process(Process-90, started)>True
<Process(Process-89, started)>True
<Process(Process-88, started)>True
<Process(Process-87, started)>True
<Process(Process-86, started)>True
<Process(Process-85, started)>True
<Process(Process-84, started)>True
<Process(Process-83, started)>True
<Process(Process-82, started)>True
<Process(Process-81, started)>True
<Process(Process-80, started)>True
<Process(Process-79, started)>True
<Process(Process-78, started)>True
<Process(Process-77, started)>True
<Process(Process-76, started)>True
<Process(Process-75, started)>True
<Process(Process-74, started)>True
<Process(Process-73, started)>True
<Process(Process-72, started)>True
<Process(Process-71, started)>True
<Process(Process-70, started)>True
<Process(Process-69, started)>True
<Process(Process-68, started)>True
<Process(Process-67, started)>True
<Process(Process-66, started)>True
<Process(Process-65, started)>True
<Process(Process-64, started)>True
<Process(Process-63, started)>True
<Process(Process-62, started)>True
<Process(Process-61, started)>True
<Process(Process-60, started)>True
<Process(Process-59, started)>True
<Process(Process-58, started)>True
<Process(Process-57, started)>True
<Process(Process-56, started)>True
<Process(Process-55, started)>True
<Process(Process-54, started)>True
<Process(Process-53, started)>True
<Process(Process-52, started)>True
<Process(Process-51, started)>True
<Process(Process-50, started)>True
<Process(Process-49, started)>True
<Process(Process-48, started)>True
<Process(Process-47, started)>True
<Process(Process-46, started)>True
<Process(Process-45, started)>True
<Process(Process-44, started)>True
<Process(Process-43, started)>True
<Process(Process-42, started)>True
<Process(Process-41, started)>True
<Process(Process-40, started)>True
<Process(Process-39, started)>True
<Process(Process-38, started)>True
<Process(Process-37, started)>True
<Process(Process-36, started)>True
<Process(Process-35, started)>True
<Process(Process-34, started)>True
<Process(Process-33, started)>True
<Process(Process-32, started)>True
<Process(Process-31, started)>True
<Process(Process-30, started)>True
<Process(Process-29, started)>True
<Process(Process-28, started)>True
<Process(Process-27, started)>True
<Process(Process-26, started)>True
<Process(Process-25, started)>True
<Process(Process-24, started)>True
<Process(Process-23, started)>True
<Process(Process-22, started)>True
<Process(Process-21, started)>True
<Process(Process-20, started)>True
<Process(Process-19, started)>True
<Process(Process-18, started)>True
<Process(Process-17, started)>True
<Process(Process-16, started)>True
<Process(Process-15, started)>True
<Process(Process-14, started)>True
<Process(Process-13, started)>True
<Process(Process-12, started)>True
<Process(Process-11, started)>True
<Process(Process-10, started)>True
<Process(Process-9, started)>True
<Process(Process-8, started)>True
<Process(Process-7, started)>True
<Process(Process-6, started)>True
<Process(Process-5, started)>True
<Process(Process-4, started)>True
<Process(Process-3, started)>True
<Process(Process-2, started)>True
<Process(Process-1, started)>True
=============================


我发现有些状态为False的,我现在的思路就是想用进程池来进行实现,但是我发现就是启动不起来,不知道什么原因,急,求大佬来解决下!!!

程序代码:
class Command(BaseCommand):

    task_dict = dict()
    INTERVAL = 60
    RS = settings.RS

    def handle(self, *args, **options):

        def _process(_processor):
            _processor.process()
        while True:
            close_old_connections()
            for token in Token.objects.all():
                key = 'token:%s' % token.value
                if not self.RS.exists(key):
                    self.RS.hset(key, 'reset', '')

            # 改写进程池启动,因为公司监控条目有很多,如果不限制启动进程的话,动则上百个进程,导致一些进程起不来,直接启动状态为stopped.
            print(1111111111111111111)
    task_dict = dict()
    INTERVAL = 60
    RS = settings.RS

    def handle(self, *args, **options):

        def _process(_processor):
            _processor.process()
        while True:
            close_old_connections()
            for token in Token.objects.all():
                key = 'token:%s' % token.value
                if not self.RS.exists(key):
                    self.RS.hset(key, 'reset', '')

            # 改写进程池启动,因为公司监控条目有很多,如果不限制启动进程的话,动则上百个进程,导致一些进程起不来,直接启动状态为stopped.
            print(1111111111111111111)
            pool = Pool(processes=10)
            resList = []
            for task in Task.objects.filter(~Q(id__in={})):  # 遍历所有的任务
                processor = TaskProcessor(task)
                print(22222222222222222)
                res = pool.apply_async(_process, args=(processor,))
                self.task_dict[task.id] = res
                print(33333333333333333)
                #resList.append(res)
            pool.close()
            pool.join()
            #for res in resList:
                #self.task_dict[res] = res
            # 原版
            # for task in Task.objects.filter(~Q(id__in=self.task_dict)):  # 第一次遍历所有的任务
            #     processor = TaskProcessor(task)
            #     p = Process(target=_process, args=(processor,))
            #     p.start()
            #     self.task_dict[task.id] = p

            with open("/home/docker/Github-Monitor/process.txt", "a") as f:
                f.write(str("===============\n"))
                f.write(str(self.task_dict))
                f.write(str("===============\n"))

            # with open("/home/docker/Github-Monitor/alive.txt", "a") as f:
            #     for task in self.task_dict:
            #         flag = self.task_dict[task].is_alive()
            #         f.write(str(self.task_dict[task]) + str(flag) + "\n")
            #     f.write("=============================\n")

            # 对于已删除的任务, 将其进程终止掉
            # task_dict_copy = self.task_dict.copy()
            # for task_id in task_dict_copy:
            #     close_old_connections()
            #     if not Task.objects.filter(id=task_id).count():
            #         # task be deleted
            #         print('terminate')
            #         self.task_dict[task_id].terminate()
            #         self.task_dict.pop(task_id)
            time.sleep(self.INTERVAL)


/home/docker/Github-Monitor/process.txt的内容如下,print的1111 222 333都有进行打印,唯独就是子进程没有启动起来

===============
{1: <multiprocessing.pool.ApplyResult object at 0x7f4d19fa8080>, 2: <multiprocessing.pool.ApplyResult object at 0x7f4d19fa8588>, 3: <multiprocessing.pool.ApplyResult object at 0x7f4d19fa8630>, 4: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3438>, 6: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb35c0>, 8: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3748>, 10: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3908>, 12: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3ac8>, 13: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3c88>, 14: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb3e48>, 15: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6048>, 16: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6208>, 17: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb63c8>, 18: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6588>, 19: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6748>, 20: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6908>, 21: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6ac8>, 22: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6c88>, 23: <multiprocessing.pool.ApplyResult object at 0x7f4d19fb6e48>, 24: <multiprocessing.pool.ApplyResult object at 0x7f4d18531048>, 25: <multiprocessing.pool.ApplyResult object at 0x7f4d18531208>, 27: <multiprocessing.pool.ApplyResult object at 0x7f4d185313c8>, 35: <multiprocessing.pool.ApplyResult object at 0x7f4d18531588>}===============
===============


求帮忙。。。。。

[此贴子已经被作者于2021-8-1 17:20编辑过]

搜索更多相关主题的帖子: Task pool Process object False 
2021-08-01 17:11
cbd666
Rank: 1
等 级:新手上路
帖 子:29
专家分:0
注 册:2019-9-26
收藏
得分:0 
是linux上面跑的,这里为什么是stopped的状态也不清楚,后面想着在task_dict中进行遍历Process的状态,把is_alive为false的进行terminate,如下尝试代码,但是发现好像没啥用

程序代码:
            # 对于已删除的任务, 将其进程终止掉
            task_dict_copy = self.task_dict.copy()
            for task_id in task_dict_copy:
                close_old_connections()
                if not task_dict_copy[task_id].is_alive():
                    print(task.__dict__[task_id],'not alive terminate')
                    self.task_dict[task_id].terminate()
                    self.task_dict.pop(task_id)
2021-08-01 17:23
快速回复:急,进程池pool apply_async 子进程无法启动
数据加载中...
 
   



关于我们 | 广告合作 | 编程中国 | 清除Cookies | TOP | 手机版

编程中国 版权所有,并保留所有权利。
Powered by Discuz, Processed in 0.035664 second(s), 10 queries.
Copyright©2004-2024, BCCN.NET, All Rights Reserved