running on jetson TX2 - resources issue

[epifanio]

I'm running in a resource allocation issue when attemting to run the code on a jetson TX2 device. The error log is:

 python stylize_image.py --input_img_path ./results/chicago.jpg \
>                         --output_img_path ./out.jpg \
>                         --model_path ./models/starry_final.ckpt
2018-09-13 20:19:03.686161: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:864] ARM64 does not support NUMA - returning NUMA node zero
2018-09-13 20:19:03.686364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 1.01GiB
2018-09-13 20:19:03.686415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-13 20:19:05.443876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-13 20:19:05.443982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2018-09-13 20:19:05.444012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2018-09-13 20:19:05.444352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 350 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
Loading up model...
Evaluating...
2018-09-13 20:19:07.947519: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 406.72MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.662035: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.725371: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 482.91MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.762340: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.824225: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 469.12MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.854860: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.916448: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 459.56MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:08.948961: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:09.011708: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 446.06MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:09.043441: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-13 20:19:19.559070: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 165.48MiB.  Current allocation summary follows.
2018-09-13 20:19:19.559179: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): 	Total Chunks: 35, Chunks in use: 35. 8.8KiB allocated for chunks. 8.8KiB in use in bin. 6.3KiB client-requested in use in bin.
2018-09-13 20:19:19.559215: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559248: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-09-13 20:19:19.559278: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559309: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559345: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): 	Total Chunks: 3, Chunks in use: 2. 42.0KiB allocated for chunks. 30.5KiB in use in bin. 30.4KiB client-requested in use in bin.
2018-09-13 20:19:19.559378: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): 	Total Chunks: 3, Chunks in use: 2. 66.2KiB allocated for chunks. 48.2KiB in use in bin. 36.0KiB client-requested in use in bin.
2018-09-13 20:19:19.559409: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): 	Total Chunks: 1, Chunks in use: 0. 36.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559458: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): 	Total Chunks: 2, Chunks in use: 2. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2018-09-13 20:19:19.559491: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): 	Total Chunks: 11, Chunks in use: 10. 1.62MiB allocated for chunks. 1.41MiB in use in bin. 1.41MiB client-requested in use in bin.
2018-09-13 20:19:19.559522: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559584: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559614: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559643: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559671: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559699: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559764: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): 	Total Chunks: 1, Chunks in use: 0. 20.69MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559797: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559826: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559860: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): 	Total Chunks: 2, Chunks in use: 1. 328.25MiB allocated for chunks. 165.48MiB in use in bin. 165.48MiB client-requested in use in bin.
2018-09-13 20:19:19.559889: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-09-13 20:19:19.559921: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 165.48MiB was 128.00MiB, Chunk State: 
2018-09-13 20:19:19.559962: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 162.76MiB | Requested Size: 496.5KiB | in_use: 0, prev:   Size: 165.48MiB | Requested Size: 165.48MiB | in_use: 1
2018-09-13 20:19:19.559993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0000 of size 1280
2018-09-13 20:19:19.560049: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0500 of size 256
2018-09-13 20:19:19.560079: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0600 of size 256
2018-09-13 20:19:19.560324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0700 of size 256
2018-09-13 20:19:19.560362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0800 of size 256
2018-09-13 20:19:19.560411: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0900 of size 256
2018-09-13 20:19:19.560455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c0a00 of size 30976
2018-09-13 20:19:19.560482: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8300 of size 256
2018-09-13 20:19:19.560505: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8400 of size 256
2018-09-13 20:19:19.560528: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8500 of size 256
2018-09-13 20:19:19.560552: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8600 of size 256
2018-09-13 20:19:19.560574: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8700 of size 256
2018-09-13 20:19:19.560597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8800 of size 256
2018-09-13 20:19:19.560620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8900 of size 256
2018-09-13 20:19:19.560643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8a00 of size 256
2018-09-13 20:19:19.560666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8b00 of size 256
2018-09-13 20:19:19.560707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8c00 of size 256
2018-09-13 20:19:19.560733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8d00 of size 256
2018-09-13 20:19:19.560756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8e00 of size 256
2018-09-13 20:19:19.560781: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c8f00 of size 256
2018-09-13 20:19:19.560804: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9000 of size 256
2018-09-13 20:19:19.560826: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9100 of size 256
2018-09-13 20:19:19.560848: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9200 of size 256
2018-09-13 20:19:19.560869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9300 of size 256
2018-09-13 20:19:19.560890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9400 of size 256
2018-09-13 20:19:19.560912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9500 of size 256
2018-09-13 20:19:19.560934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9600 of size 256
2018-09-13 20:19:19.560955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9700 of size 256
2018-09-13 20:19:19.560978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9800 of size 256
2018-09-13 20:19:19.561033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9900 of size 256
2018-09-13 20:19:19.561058: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9a00 of size 256
2018-09-13 20:19:19.561080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9b00 of size 256
2018-09-13 20:19:19.561102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9c00 of size 256
2018-09-13 20:19:19.561124: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9d00 of size 256
2018-09-13 20:19:19.561155: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10c9e00 of size 256
2018-09-13 20:19:19.561178: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfc10c9f00 of size 11776
2018-09-13 20:19:19.561203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10ccd00 of size 15616
2018-09-13 20:19:19.561227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10d0a00 of size 256
2018-09-13 20:19:19.561248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10d0b00 of size 15616
2018-09-13 20:19:19.561270: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfc10d4800 of size 18432
2018-09-13 20:19:19.561293: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10d9000 of size 18432
2018-09-13 20:19:19.561316: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfc10dd800 of size 36864
2018-09-13 20:19:19.561337: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10e6800 of size 256
2018-09-13 20:19:19.561361: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc10e6900 of size 147456
2018-09-13 20:19:19.561393: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc110a900 of size 73728
2018-09-13 20:19:19.561424: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc111c900 of size 147456
2018-09-13 20:19:19.561447: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1140900 of size 147456
2018-09-13 20:19:19.561493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1164900 of size 147456
2018-09-13 20:19:19.561517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1188900 of size 147456
2018-09-13 20:19:19.561539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc11ac900 of size 147456
2018-09-13 20:19:19.561560: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc11d0900 of size 147456
2018-09-13 20:19:19.561582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc11f4900 of size 147456
2018-09-13 20:19:19.561604: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1218900 of size 147456
2018-09-13 20:19:19.561626: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfc123c900 of size 221184
2018-09-13 20:19:19.561647: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1272900 of size 147456
2018-09-13 20:19:19.561669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc1296900 of size 73728
2018-09-13 20:19:19.561691: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfc12a8900 of size 21690368
2018-09-13 20:19:19.561714: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0xfc2758100 of size 173522944
2018-09-13 20:19:19.561736: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0xfcccd4100 of size 170667776
2018-09-13 20:19:19.561757: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size: 
2018-09-13 20:19:19.561783: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 35 Chunks of size 256 totalling 8.8KiB
2018-09-13 20:19:19.561809: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2018-09-13 20:19:19.561843: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 15616 totalling 30.5KiB
2018-09-13 20:19:19.561877: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 18432 totalling 18.0KiB
2018-09-13 20:19:19.561904: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 30976 totalling 30.2KiB
2018-09-13 20:19:19.561929: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 73728 totalling 144.0KiB
2018-09-13 20:19:19.561956: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 147456 totalling 1.41MiB
2018-09-13 20:19:19.561982: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 173522944 totalling 165.48MiB
2018-09-13 20:19:19.562007: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 167.12MiB
2018-09-13 20:19:19.562040: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: 
Limit:                   367882240
InUse:                   175235840
MaxInUse:                197812736
NumAllocs:                     289
MaxAllocSize:            173522944

2018-09-13 20:19:19.562119: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *_____************************************************______________________________________________
2018-09-13 20:19:19.562196: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[1,32,952,1424] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "stylize_image.py", line 75, in <module>
    img_out = sess.run(Y, feed_dict={X: img_4d})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,32,952,1424] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[Node: img_t_net/upsample_1/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](img_t_net/upsample_1/resize_images/ResizeNearestNeighbor, PermConstNHWCToNCHW-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: img_t_net/output/_99 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_548_img_t_net/output", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Have you any advice on how (if possible) tu run faststyle on a Jetson TX2?

[ghwatson]

Unfortunately you're on your own here as I've never used that device. Looks like it should have enough memory (8GB). I was using a GTX 970 when I made this repo which is a 4GB card.