ColabFold

RuntimeError: Internal: Failed to complete all kernels launched on stream 0x55f45bedc140: stream did not block host until done; was already in an error state
2021-09-22 11:57:31.482813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482861: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482879: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482884: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482888: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482897: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482951: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482959: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482968: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482991: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483014: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483027: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483078: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483102: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483126: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483131: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483136: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483159: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483185: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483204: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483209: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483219: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483242: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483252: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483287: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483309: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483314: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483318: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483349: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483382: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483401: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483406: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483415: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483439: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483450: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483468: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483473: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483505: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483511: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483515: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483533: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483538: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483543: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483548: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483592: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483618: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483641: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483665: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483674: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483684: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483692: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483707: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483713: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483743: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483753: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483771: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483776: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483786: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483804: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483814: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483818: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483851: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483866: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483885: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483892: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483899: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483904: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483908: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483918: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483940: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483950: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483973: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483991: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484007: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484012: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484017: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484024: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484040: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484045: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484050: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484068: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484072: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484100: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484104: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484114: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484133: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484143: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484148: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484166: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484180: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484187: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484203: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484208: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484213: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484242: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484246: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484254: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484270: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484293: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484302: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484312: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484335: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484345: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484368: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484378: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484392: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484397: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484407: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484411: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484434: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484440: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484444: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484467: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484472: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484477: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484495: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484517: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484523: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484552: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484586: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484591: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484595: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484604: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484616: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484620: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484634: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484652: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484657: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484667: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484688: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484693: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484749: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484782: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484793: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484821: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484856: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484904: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484909: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484916: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484932: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484937: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484942: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484960: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484974: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484994: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484998: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485008: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485041: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485070: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485074: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485097: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485102: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485107: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485130: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485140: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485147: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485163: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485172: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485180: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485195: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485205: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485230: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485235: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485255: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485264: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485288: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485293: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485297: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485317: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485327: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485334: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485365: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485375: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485398: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485425: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.508920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1039] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered :: *** Begin stack trace ***

    PyDict_SetItem
    _PyModule_ClearDict
    PyImport_Cleanup
    Py_FinalizeEx

    _Py_UnixMain
    __libc_start_main

*** End stack trace ***

2021-09-22 11:57:31.509012: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_executable.cc:99] Check failed: pair.first->SynchronizeAllActivity() 

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 26%   43C    P0     1W / 215W |      0MiB /  7973MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

[colabfold]# colabfold-conda/bin/python3.7 runner.py
2021-09-22 13:34:19.030941: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-22 13:34:19.653032: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-09-22 13:34:20.272119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 13:34:20.272380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 computeCapability: 7.5
coreClock: 1.71GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.23GiB/s
2021-09-22 13:34:20.272399: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-09-22 13:34:20.273838: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-09-22 13:34:20.273864: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-09-22 13:34:20.274345: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-09-22 13:34:20.274465: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-09-22 13:34:20.275972: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-09-22 13:34:20.276338: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-09-22 13:34:20.276408: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-09-22 13:34:20.276460: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 13:34:20.276721: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-22 13:34:20.276946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
ColabFold on Linux
2021-09-22 13:34:20.902851: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-22 13:34:20.903675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-22 13:34:20.903687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
1 Physical GPUs, 0 Logical GPUs
homooligomer: '1'
total_length: '253'
working_directory: 'prediction_7NC5_1_a7cd8'
running mmseqs2
  0%|          | 0/5 [elapsed: 00:00 remaining: ?]
2021-09-22 13:34:23.946842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-22 13:34:23.946868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
2021-09-22 13:34:23.994757: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3600000000 Hz
2021-09-22 13:35:16.802536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1155] failed to enqueue async memcpy from device to host: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; host dst: 0x7fff144d418f; GPU src: 0x7f9f9221d000; size: 1=0x1
2021-09-22 13:35:16.802628: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2040] Execution of replica 0 failed: Internal: Failed to complete all kernels launched on stream 0x5646eaa04fc0: stream did not block host until done; was already in an error state
Traceback (most recent call last):
  File "runner.py", line 675, in <module>
    prediction_result, (r, t) = cf.to(model_runner.predict(processed_feature_dict, random_seed=seed),"cpu")
  File "/home/fpe/colabfold/alphafold/model/model.py", line 134, in predict
    result, recycles = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/_src/api.py", line 415, in cache_miss
    donated_invars=donated_invars, inline=inline)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/core.py", line 1616, in bind
    return call_bind(self, fun, *args, **params)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/core.py", line 1607, in call_bind
    outs = primitive.process(top_trace, fun, tracers, params)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/core.py", line 1619, in process
    return trace.process_call(self, fun, tracers, params)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/core.py", line 615, in process_call
    return primitive.impl(f, *tracers, **params)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/interpreters/xla.py", line 623, in _xla_call_impl
    out = compiled_fun(*args)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/interpreters/xla.py", line 913, in _execute_compiled
    out_bufs = compiled.execute(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: RuntimeError: Internal: Failed to complete all kernels launched on stream 0x5646eaa04fc0: stream did not block host until done; was already in an error state

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "runner.py", line 675, in <module>
    prediction_result, (r, t) = cf.to(model_runner.predict(processed_feature_dict, random_seed=seed),"cpu")
  File "/home/fpe/colabfold/alphafold/model/model.py", line 134, in predict
    result, recycles = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
  File "/home/fpe/colabfold/colabfold-conda/lib/python3.7/site-packages/jax/interpreters/xla.py", line 913, in _execute_compiled
    out_bufs = compiled.execute(input_bufs)
RuntimeError: Internal: Failed to complete all kernels launched on stream 0x5646eaa04fc0: stream did not block host until done; was already in an error state
2021-09-22 13:35:17.161089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161139: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161163: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161167: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161172: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161175: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161228: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161242: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161246: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161255: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161261: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161269: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161278: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161292: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161296: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161300: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161304: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161308: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161329: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161333: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161337: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161351: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161379: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161385: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161390: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161398: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161403: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161408: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161411: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161416: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161420: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161427: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161442: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161448: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161453: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161472: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161476: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161482: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161487: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161500: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161510: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161520: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161535: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161544: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161552: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161557: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161565: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161601: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161611: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161642: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161646: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161651: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161654: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161661: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161676: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161682: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161686: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161695: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161701: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161706: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161710: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161716: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161720: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161728: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161743: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161749: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161753: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161772: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161776: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161782: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161786: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161803: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161807: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161815: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161834: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161839: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161848: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161856: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161867: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161872: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161882: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161895: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161905: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161915: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161928: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161933: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161938: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161948: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161973: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.161983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.161990: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.161996: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162001: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162006: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162011: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162016: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162038: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162044: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162048: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162056: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162071: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162077: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162081: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162090: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162105: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162111: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162115: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162123: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162148: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162166: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162171: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162181: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162204: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162210: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162214: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162234: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162238: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162248: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162262: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162271: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162277: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162281: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162288: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162300: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162304: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162314: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162337: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162347: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162371: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162381: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162394: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162404: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162414: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162427: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162432: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162436: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162446: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162454: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162460: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162469: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162475: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162479: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162499: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162503: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162513: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162521: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162531: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162546: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162562: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162571: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162577: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162589: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162605: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162615: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162623: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162638: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162648: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162671: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162681: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162689: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162694: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162704: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162714: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162732: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162737: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162742: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162747: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162771: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162776: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162781: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162803: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162813: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162831: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162835: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162841: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162845: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162853: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162859: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162868: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162895: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162904: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162914: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162928: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162932: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162937: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162942: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162947: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162961: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162966: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162970: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162980: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.162987: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.162993: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.162998: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163012: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163031: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163036: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163046: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163064: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163068: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163074: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163078: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163086: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163101: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163111: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163125: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163130: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163134: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163140: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163145: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163159: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163168: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163174: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163178: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163185: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163201: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163211: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163233: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163243: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163257: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163261: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163266: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163290: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163299: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163309: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163317: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163332: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163337: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163342: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163364: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163374: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163397: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163403: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163407: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163415: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163428: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163433: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163450: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163465: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163476: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163483: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163495: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163517: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.163523: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x5646e9426aa0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 13:35:17.163543: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 13:35:17.163553: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 13:35:17.186678: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1039] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered :: *** Begin stack trace ***

    PyDict_SetItem
    _PyModule_ClearDict
    PyImport_Cleanup
    Py_FinalizeEx

    _Py_UnixMain
    __libc_start_main

*** End stack trace ***

2021-09-22 13:35:17.186759: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_executable.cc:99] Check failed: pair.first->SynchronizeAllActivity()