ColabFold

RuntimeError: Internal: Failed to complete all kernels launched on stream 0x55f45bedc140: stream did not block host until done; was already in an error state
2021-09-22 11:57:31.482813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482861: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482879: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482884: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482888: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482897: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482951: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482959: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482968: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.482983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.482991: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.482997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483014: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483027: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483078: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483102: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483126: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483131: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483136: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483159: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483185: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483204: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483209: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483219: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483242: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483252: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483287: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483309: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483314: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483318: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483349: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483382: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483401: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483406: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483415: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483439: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483450: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483468: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483473: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483505: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483511: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483515: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483533: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483538: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483543: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483548: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483592: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483618: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483641: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483665: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483674: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483684: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483692: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483707: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483713: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483743: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483753: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483771: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483776: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483786: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483804: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483814: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483818: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483851: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483866: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483885: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483892: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483899: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483904: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483908: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483918: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483940: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483950: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483973: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.483983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.483991: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.483997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484007: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484012: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484017: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484024: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484040: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484045: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484050: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484068: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484072: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484100: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484104: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484114: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484133: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484143: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484148: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484166: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484180: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484187: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484203: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484208: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484213: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484242: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484246: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484254: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484270: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484293: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484302: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484312: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484335: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484345: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484368: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484378: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484392: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484397: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484407: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484411: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484434: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484440: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484444: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484467: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484472: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484477: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484495: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484517: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484523: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484552: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484586: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484591: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484595: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484604: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484616: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484620: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484634: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484652: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484657: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484667: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484688: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484693: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484749: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484782: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484793: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484821: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484856: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484904: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484909: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484916: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484932: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484937: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484942: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484960: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484974: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.484983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.484989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484994: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.484998: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485008: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485041: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485070: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485074: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485097: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485102: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485107: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485130: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485140: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485147: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485163: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485172: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485180: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485195: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485205: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485230: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485235: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485255: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485264: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485288: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485293: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485297: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485317: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485327: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485334: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485365: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485375: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.485388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485398: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:692] could not allocate CUDA stream for context 0x55f45a89d1e0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2021-09-22 11:57:31.485425: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2021-09-22 11:57:31.485433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:611] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2021-09-22 11:57:31.508920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1039] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered :: *** Begin stack trace ***

    PyDict_SetItem
    _PyModule_ClearDict
    PyImport_Cleanup
    Py_FinalizeEx

    _Py_UnixMain
    __libc_start_main

*** End stack trace ***

2021-09-22 11:57:31.509012: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_executable.cc:99] Check failed: pair.first->SynchronizeAllActivity() 

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 26%   43C    P0     1W / 215W |      0MiB /  7973MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)