'[jira] [Created] (MESOS-10233) containers were not cleaned up properly and left running.'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mesos-issues
Subject:    [jira] [Created] (MESOS-10233) containers were not cleaned up properly and left running.
From:       "Tan N. Le (Jira)" <jira () apache ! org>
Date:       2021-10-29 14:45:00
Message-ID: JIRA.13409136.1635518668000.1200787.1635518700540 () Atlassian ! JIRA
[Download RAW message or body]

Tan N. Le created MESOS-10233:
---------------------------------

             Summary: containers were not cleaned up properly and left running.
                 Key: MESOS-10233
                 URL: https://issues.apache.org/jira/browse/MESOS-10233
             Project: Mesos
          Issue Type: Task
         Environment: aurora-scheduler 0.25.0

mesos 1.11.0

executor plugin: DCE [https://github.com/paypal/dce-go]  based on mesos-go v0.002

  
            Reporter: Tan N. Le


We observe that tasks were in STARTING and mesos tried to killed and cleaned them up \
dueo to OOM.

however, cgroup freezer files are not there and it assumes the containers are being \
cleaned.

the containers left running but the tasks reported lost in aurora/mesos.

  

================

aurora logs
I1026 05:16:55.886 [TaskEventBatchWorker, StateMachine] \
-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition INIT -> PENDING I1026 \
05:16:55.984 [TaskGroupBatchWorker, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state \
machine transition PENDING -> ASSIGNED I1026 05:16:55.984 [TaskGroupBatchWorker, \
TaskAssignerImpl] Offer on agent \
gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com (id \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307) is being assigned task for \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728. I1026 \
05:16:57.402 [Thread-1715969, MesosCallbackHandler$MesosCallbackHandlerImpl] Received \
status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in state \
TASK_STARTING from SOURCE_EXECUTOR I1026 05:16:57.402 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 \
state machine transition ASSIGNED -> STARTING W1026 05:18:03.376 [Thread-1717148, \
MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor \
compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 \
on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 with status -1 I1026 05:18:03.377 \
[Thread-1717149, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status \
update for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 \
in state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal \
executor termination: Failed to kill all processes in the container: Timed out after \
1mins I1026 05:18:03.390 [TaskStatusHandlerImpl, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state \
machine transition STARTING -> FAILED I1026 05:18:03.390 [TaskStatusHandlerImpl, \
StateManagerImpl] Task being rescheduled: \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 I1026 \
05:21:08.948 [Thread-1720928, MesosCallbackHandler$MesosCallbackHandlerImpl] Received \
status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in state \
TASK_RUNNING from SOURCE_EXECUTOR I1026 05:21:08.949 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 \
state machine transition FAILED -> RUNNING (not allowed) I1026 05:21:08.950 \
[TaskStatusHandlerImpl, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state \
machine transition FAILED -> LOST (not allowed)

=================

mesos-master logs

I1026 05:16:55.991168 29973 master.cpp:3873] Adding executor \
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' \
with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of framework \
9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:16:55.991324 29973 \
master.cpp:3899] Adding task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 with \
resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100; mem(allocated: \
aurora):4096; ports(allocated: aurora):[10020-10020, 10076-10076, 10137-10137, \
10139-10139, 10150-10150] of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 \
(Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on \
agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:16:56.090255 29973 \
master.cpp:5035] Launching task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 with resources \
[\{"allocation_info":{"role":"aurora"},"name":"cpus","scalar":\{"value":0.9},"type":"S \
CALAR"},\{"allocation_info":{"role":"aurora"},"name":"disk","scalar":\{"value":100.0}, \
"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"mem","scalar":\{"value" \
:4096.0},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"ports","ranges \
":\{"range":[{"begin":10020,"end":10020},\{"begin":10076,"end":10076},\{"begin":10137, \
"end":10137},\{"begin":10139,"end":10139},\{"begin":10150,"end":10150}]},"type":"RANGES"}] \
on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) on new executor I1026 \
05:16:57.401727 29979 master.cpp:8160] Status update TASK_STARTING (Status UUID: \
9d316a48-3656-11ec-8d10-42010ab432d2) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:16:57.401800 29979 \
master.cpp:8217] Forwarding status update TASK_STARTING (Status UUID: \
9d316a48-3656-11ec-8d10-42010ab432d2) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 I1026 05:16:57.401979 29979 \
master.cpp:11195] Updating the state of task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state: TASK_STARTING, \
status update state: TASK_STARTING) I1026 05:16:57.409750 29990 master.cpp:5870] \
Processing ACKNOWLEDGE call for status 9d316a48-3656-11ec-8d10-42010ab432d2 for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307

I1026 05:18:03.375757 29982 master.cpp:8497] Executor \
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' \
of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com): wait status -1 I1026 \
05:18:03.375849 29982 master.cpp:11335] Removing executor \
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' \
with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of framework \
9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:18:03.376816 29990 \
master.cpp:8160] Status update TASK_FAILED (Status UUID: \
7fbe4d73-2eed-4713-8774-8299f3df9736) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:18:03.376888 29990 \
master.cpp:8217] Forwarding status update TASK_FAILED (Status UUID: \
7fbe4d73-2eed-4713-8774-8299f3df9736) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 I1026 05:18:03.377179 29990 \
master.cpp:11195] Updating the state of task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state: TASK_FAILED, \
status update state: TASK_FAILED) I1026 05:18:03.505921 29974 master.cpp:5870] \
Processing ACKNOWLEDGE call for status 7fbe4d73-2eed-4713-8774-8299f3df9736 for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 I1026 05:18:03.505990 29974 \
master.cpp:11295] Removing task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 with \
resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100; mem(allocated: \
aurora):4096; ports(allocated: aurora):[10020-10020, 10076-10076, 10137-10137, \
10139-10139, 10150-10150] of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on \
agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:21:08.947870 29968 \
master.cpp:8160] Status update TASK_RUNNING (Status UUID: \
332069b9-3657-11ec-8d10-42010ab432d2) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:21:08.947954 29968 \
master.cpp:8217] Forwarding status update TASK_RUNNING (Status UUID: \
332069b9-3657-11ec-8d10-42010ab432d2) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 W1026 05:21:08.948144 29968 \
master.cpp:8190] Could not lookup task for status update TASK_RUNNING (Status UUID: \
332069b9-3657-11ec-8d10-42010ab432d2) for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 \
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) I1026 05:21:08.949649 29980 \
master.cpp:5696] Processing KILL call for task \
'mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 W1026 \
05:21:08.949703 29980 master.cpp:5703] Cannot kill task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 because it is \
unknown; performing reconciliation I1026 05:21:08.949753 29980 master.cpp:9140] \
Sending explicit reconciliation state TASK_LOST for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 I1026 \
05:21:08.949921 29980 master.cpp:5870] Processing ACKNOWLEDGE call for status \
332069b9-3657-11ec-8d10-42010ab432d2 for task \
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of \
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at \
scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307

==================
mesos-slave logs

E1026 05:17:03.372925 849 memory.cpp:665] Failed to read 'memory.limit_in_bytes': No \
such file or directory E1026 05:17:03.373068 849 memory.cpp:674] Failed to read \
'memory.max_usage_in_bytes': No such file or directory E1026 05:17:03.373138 849 \
memory.cpp:686] Failed to read 'memory.stat': No such file or directory W1026 \
05:17:03.373395 849 linux_launcher.cpp:600] Couldn't find freezer cgroup for \
container f9e69838-169e-4837-be5e-11ef4860bb01 so assuming partially destroyed W1026 \
05:17:03.373773 848 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.375150 839 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.377418 841 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.381808 832 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.390095 829 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.406415 830 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.438736 843 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.503955 836 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.632373 846 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:03.888777 845 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:04.401167 826 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:05.425781 838 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:07.475064 827 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:11.571998 847 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:19.764526 847 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again W1026 05:17:36.149070 841 cgroups.cpp:294] Removal of cgroup \
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, \
will try again E1026 05:18:03.375078 835 slave.cpp:7085] Termination of executor \
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' \
of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 failed: Failed to kill all \
processes in the container: Timed out after 1mins


I1026 05:16:39.505 [qtp1387308322-1609115, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 state \
machine transition RUNNING -> KILLING I1026 05:16:52.192 [Thread-1715877, \
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 in state \
TASK_KILLED from SOURCE_EXECUTOR I1026 05:16:52.192 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 \
state machine transition KILLING -> KILLED I1026 05:16:52.243 [TaskEventBatchWorker, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
state machine transition INIT -> PENDING I1026 05:16:52.300 [TaskGroupBatchWorker, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
state machine transition PENDING -> ASSIGNED I1026 05:16:52.300 \
[TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent \
gpma773817.gpf-prod.us-central1.gcp.dev.paypalinc.com (id \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S470) is being assigned task for \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649. I1026 \
05:16:53.643 [Thread-1715899, MesosCallbackHandler$MesosCallbackHandlerImpl] Received \
status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in state \
TASK_STARTING from SOURCE_EXECUTOR I1026 05:16:53.643 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
state machine transition ASSIGNED -> STARTING W1026 05:18:02.991 [Thread-1717141, \
MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor \
compose-mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S470 with status -1 I1026 05:18:02.991 \
[Thread-1717142, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status \
update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
in state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal \
executor termination: Failed to kill all processes in the container: Timed out after \
1mins I1026 05:18:03.094 [TaskStatusHandlerImpl, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state \
machine transition STARTING -> FAILED I1026 05:18:03.095 [TaskStatusHandlerImpl, \
StateManagerImpl] Task being rescheduled: \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 I1026 \
05:18:03.095 [TaskStatusHandlerImpl, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 state \
machine transition INIT -> PENDING I1026 05:18:03.227 [TaskGroupBatchWorker, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 \
state machine transition PENDING -> ASSIGNED I1026 05:18:03.227 \
[TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent \
gpma778297.gpf-prod.us-central1.gcp.dev.paypalinc.com (id \
a76961ab-bba0-46e5-ae7b-b234057b7a33-S609) is being assigned task for \
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889. I1026 \
05:18:04.520 [Thread-1717191, MesosCallbackHandler$MesosCallbackHandlerImpl] Received \
status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in state \
TASK_STARTING from SOURCE_EXECUTOR I1026 05:18:04.520 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 \
state machine transition ASSIGNED -> STARTING I1026 05:20:52.246 [Thread-1720630, \
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in state \
TASK_RUNNING from SOURCE_EXECUTOR I1026 05:20:52.246 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 \
state machine transition FAILED -> RUNNING (not allowed) I1026 05:20:52.248 \
[TaskStatusHandlerImpl, StateMachine] \
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state \
machine transition FAILED -> LOST (not allowed) I1026 05:22:50.010 [Thread-1723190, \
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task \
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in state \
TASK_RUNNING from SOURCE_EXECUTOR I1026 05:22:50.010 [TaskStatusHandlerImpl, \
StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 \
state machine transition STARTING -> RUNNING



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic