[task 2024-01-11T21:48:51.834Z] [2024-01-11 21:48:51] [Trainer] [INFO] Reading original for epoch 71
[task 2024-01-11T21:48:54.660Z] [2024-01-11 21:48:54] [Trainer] [INFO] Reading original for epoch 72
[task 2024-01-11T21:48:57.467Z] [2024-01-11 21:48:57] [Trainer] [INFO] Reading original for epoch 73
[task 2024-01-11T21:49:00.297Z] [2024-01-11 21:49:00] [Trainer] [INFO] Reading original for epoch 74
[task 2024-01-11T21:49:03.151Z] [2024-01-11 21:49:03] [Trainer] [INFO] Reading original for epoch 75
[task 2024-01-11T21:49:05.948Z] [2024-01-11 21:49:05] [Trainer] [INFO] Reading original for epoch 76
[task 2024-01-11T21:49:08.796Z] [2024-01-11 21:49:08] [Trainer] [INFO] Reading original for epoch 77
[task 2024-01-11T21:49:11.611Z] [2024-01-11 21:49:11] [Trainer] [INFO] Reading original for epoch 78
[task 2024-01-11T21:49:14.471Z] [2024-01-11 21:49:14] [Trainer] [INFO] Reading original for epoch 79
[task 2024-01-11T21:49:17.293Z] [2024-01-11 21:49:17] [Trainer] [INFO] Reading original for epoch 80
[task 2024-01-11T21:49:20.112Z] [2024-01-11 21:49:20] [Trainer] [INFO] Reading original for epoch 81
[task 2024-01-11T21:49:22.929Z] [2024-01-11 21:49:22] [Trainer] [INFO] Reading original for epoch 82
[task 2024-01-11T21:49:25.775Z] [2024-01-11 21:49:25] [Trainer] [INFO] Reading original for epoch 83
[task 2024-01-11T21:49:28.584Z] [2024-01-11 21:49:28] [Trainer] [INFO] Reading original for epoch 84
[task 2024-01-11T21:49:31.452Z] [2024-01-11 21:49:31] [Trainer] [INFO] Reading original for epoch 85
[task 2024-01-11T21:49:34.315Z] [2024-01-11 21:49:34] [Trainer] [INFO] Reading original for epoch 86
[task 2024-01-11T21:49:37.173Z] [2024-01-11 21:49:37] [Trainer] [INFO] Reading original for epoch 87
[task 2024-01-11T21:49:40.022Z] [2024-01-11 21:49:40] [Trainer] [INFO] Reading original for epoch 88
[task 2024-01-11T21:49:42.849Z] [2024-01-11 21:49:42] [Trainer] [INFO] Reading original for epoch 89
[task 2024-01-11T21:49:45.712Z] [2024-01-11 21:49:45] [Trainer] [INFO] Reading original for epoch 90
[task 2024-01-11T21:49:48.537Z] [2024-01-11 21:49:48] [Trainer] [INFO] Reading original for epoch 91
[task 2024-01-11T21:49:51.463Z] [2024-01-11 21:49:51] [Trainer] [INFO] Reading original for epoch 92
[task 2024-01-11T21:49:54.352Z] [2024-01-11 21:49:54] [Trainer] [INFO] Reading original for epoch 93
[task 2024-01-11T21:49:57.229Z] [2024-01-11 21:49:57] [Trainer] [INFO] Reading original for epoch 94
[task 2024-01-11T21:50:36.226Z] [2024-01-11 21:50:36] [training] Batches are processed as 1 process(es) x 4 devices/process
[task 2024-01-11T21:50:36.230Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu0
[task 2024-01-11T21:50:36.230Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu1
[task 2024-01-11T21:50:36.230Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu2
[task 2024-01-11T21:50:36.230Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu3
[task 2024-01-11T21:50:36.278Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu0
[task 2024-01-11T21:50:36.338Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu1
[task 2024-01-11T21:50:36.380Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu2
[task 2024-01-11T21:50:36.403Z] [2024-01-11 21:50:36] [memory] Reserving 34 MB, device gpu3
[task 2024-01-11T21:50:36.490Z] [2024-01-11 21:50:36] [Trainer] [INFO] Reading original for epoch 95
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] Allocating memory for general optimizer shards
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] [memory] Reserving 8 MB, device gpu2
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] [memory] Reserving 8 MB, device gpu3
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] Parameter type float32, optimization type float32, casting types false
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] [memory] Reserving 8 MB, device gpu1
[task 2024-01-11T21:50:36.932Z] [2024-01-11 21:50:36] [memory] Reserving 8 MB, device gpu0
[task 2024-01-11T21:50:36.935Z] [2024-01-11 21:50:36] Allocating memory for Adam-specific shards
[task 2024-01-11T21:50:36.935Z] [2024-01-11 21:50:36] [memory] Reserving 17 MB, device gpu1
[task 2024-01-11T21:50:36.936Z] [2024-01-11 21:50:36] [memory] Reserving 17 MB, device gpu3
[task 2024-01-11T21:50:36.937Z] [2024-01-11 21:50:36] [memory] Reserving 17 MB, device gpu2
[task 2024-01-11T21:50:36.937Z] [2024-01-11 21:50:36] [memory] Reserving 17 MB, device gpu0
[task 2024-01-11T21:50:36.941Z] [2024-01-11 21:50:36] Ep. 1 : Up. 1 : Sen. 2,102 : Cost 1.77846706 : Time 316.27s : 212.68 words/s : gNorm 30.8682 : L.r. 1.8750e-08
[task 2024-01-11T21:50:37.479Z] [2024-01-11 21:50:37] Ep. 1 : Up. 2 : Sen. 3,622 : Cost 0.30264086 : Time 0.54s : 470112.76 words/s : gNorm 28.6664 : L.r. 3.7500e-08
[task 2024-01-11T21:50:37.887Z] [2024-01-11 21:50:37] Ep. 1 : Up. 3 : Sen. 5,422 : Cost 0.45938283 : Time 0.41s : 653030.20 words/s : gNorm 33.3493 : L.r. 5.6250e-08
[task 2024-01-11T21:50:38.248Z] [2024-01-11 21:50:38] Ep. 1 : Up. 4 : Sen. 7,070 : Cost 0.39801216 : Time 0.36s : 707604.62 words/s : gNorm 31.9421 : L.r. 7.5000e-08
[task 2024-01-11T21:50:38.631Z] [2024-01-11 21:50:38] Ep. 1 : Up. 5 : Sen. 10,958 : Cost 0.69845533 : Time 0.38s : 721487.36 words/s : gNorm 32.2173 : L.r. 9.3750e-08
[task 2024-01-11T21:50:38.923Z] [2024-01-11 21:50:38] Ep. 1 : Up. 6 : Sen. 16,414 : Cost 0.94875395 : Time 0.29s : 112084.51 words/s : gNorm 30.5714 : L.r. 1.1250e-07
[task 2024-01-11T21:50:39.232Z] [2024-01-11 21:50:39] [Trainer] [INFO] Reading original for epoch 96
[task 2024-01-11T21:50:39.298Z] [2024-01-11 21:50:39] Ep. 1 : Up. 7 : Sen. 18,062 : Cost 0.29143503 : Time 0.38s : 399443.53 words/s : gNorm 30.2884 : L.r. 1.3125e-07
[task 2024-01-11T21:50:39.711Z] [2024-01-11 21:50:39] Ep. 1 : Up. 8 : Sen. 19,582 : Cost 0.29566261 : Time 0.41s : 357319.48 words/s : gNorm 29.4513 : L.r. 1.5000e-07
[task 2024-01-11T21:50:40.208Z] [2024-01-11 21:50:40] Ep. 1 : Up. 9 : Sen. 20,798 : Cost 0.27952152 : Time 0.50s : 439725.79 words/s : gNorm 29.7565 : L.r. 1.6875e-07
[task 2024-01-11T21:50:40.551Z] [2024-01-11 21:50:40] Ep. 1 : Up. 10 : Sen. 22,014 : Cost 0.31261924 : Time 0.34s : 326578.53 words/s : gNorm 29.7323 : L.r. 1.8750e-07
[task 2024-01-11T21:50:40.910Z] [2024-01-11 21:50:40] Ep. 1 : Up. 11 : Sen. 24,406 : Cost 0.44481000 : Time 0.36s : 765265.63 words/s : gNorm 30.3295 : L.r. 2.0625e-07
[task 2024-01-11T21:50:41.290Z] [2024-01-11 21:50:41] Ep. 1 : Up. 12 : Sen. 25,926 : Cost 0.36008400 : Time 0.38s : 644427.06 words/s : gNorm 30.2337 : L.r. 2.2500e-07
[task 2024-01-11T21:50:41.746Z] [2024-01-11 21:50:41] Ep. 1 : Up. 13 : Sen. 27,142 : Cost 0.26075512 : Time 0.46s : 469013.24 words/s : gNorm 29.2062 : L.r. 2.4375e-07
[task 2024-01-11T21:50:41.920Z] [2024-01-11 21:50:41] [Trainer] [INFO] Reading original for epoch 97
[task 2024-01-11T21:50:42.251Z] [2024-01-11 21:50:42] Ep. 1 : Up. 14 : Sen. 28,942 : Cost 0.34564367 : Time 0.50s : 522046.02 words/s : gNorm 29.5999 : L.r. 2.6250e-07
[task 2024-01-11T21:50:42.505Z] [2024-01-11 21:50:42] Ep. 1 : Up. 15 : Sen. 30,350 : Cost 0.39673856 : Time 0.25s : 182858.53 words/s : gNorm 29.2041 : L.r. 2.8125e-07
[task 2024-01-11T21:50:42.778Z] [2024-01-11 21:50:42] Ep. 1 : Up. 16 : Sen. 32,318 : Cost 0.50238973 : Time 0.27s : 21579.31 words/s : gNorm 28.9746 : L.r. 3.0000e-07
[task 2024-01-11T21:50:42.979Z] [2024-01-11 21:50:42] Ep. 1 : Up. 17 : Sen. 33,622 : Cost 0.42501819 : Time 0.20s : 211496.14 words/s : gNorm 29.1243 : L.r. 3.1875e-07
[task 2024-01-11T21:50:43.234Z] [2024-01-11 21:50:43] Ep. 1 : Up. 18 : Sen. 39,078 : Cost 1.07669222 : Time 0.25s : 128556.91 words/s : gNorm 28.8100 : L.r. 3.3750e-07
[task 2024-01-11T21:50:43.652Z] [2024-01-11 21:50:43] Ep. 1 : Up. 19 : Sen. 40,294 : Cost 0.37256432 : Time 0.42s : 448137.11 words/s : gNorm 29.5777 : L.r. 3.5625e-07
[task 2024-01-11T21:50:44.537Z] [2024-01-11 21:50:44] [Trainer] [INFO] Reading original for epoch 98
[task 2024-01-11T21:50:47.269Z] [2024-01-11 21:50:47] [Trainer] [INFO] Reading original for epoch 99
[task 2024-01-11T21:50:49.950Z] [2024-01-11 21:50:49] [Trainer] [INFO] Reading original for epoch 100
[task 2024-01-11T21:50:52.660Z] [2024-01-11 21:50:52] [Trainer] [INFO] Reading original for epoch 101
[task 2024-01-11T21:50:55.521Z] [2024-01-11 21:50:55] [Trainer] [INFO] Reading original for epoch 102
[task 2024-01-11T21:50:58.236Z] [2024-01-11 21:50:58] [Trainer] [INFO] Reading original for epoch 103
[task 2024-01-11T21:50:59.602Z] [2024-01-11 21:50:59] Ep. 1 : Up. 20 : Sen. 41,510 : Cost 0.27730182 : Time 15.95s : 14977.73 words/s : gNorm 29.6449 : L.r. 3.7500e-07
[task 2024-01-11T21:51:00.203Z] [2024-01-11 21:51:00] Ep. 1 : Up. 21 : Sen. 42,814 : Cost 0.24364506 : Time 0.60s : 408065.76 words/s : gNorm 29.5220 : L.r. 3.9375e-07
[task 2024-01-11T21:51:00.484Z] [2024-01-11 21:51:00] Ep. 1 : Up. 22 : Sen. 45,206 : Cost 0.58817548 : Time 0.28s : 25609.01 words/s : gNorm 29.3477 : L.r. 4.1250e-07
[task 2024-01-11T21:51:00.627Z] [2024-01-11 21:51:00] Ep. 1 : Up. 23 : Sen. 46,727 : Cost 0.86194068 : Time 0.14s : 602949.20 words/s : gNorm 29.8930 : L.r. 4.3125e-07
[task 2024-01-11T21:51:00.879Z] [2024-01-11 21:51:00] Ep. 1 : Up. 24 : Sen. 48,031 : Cost 0.38329166 : Time 0.25s : 15552.37 words/s : gNorm 29.6691 : L.r. 4.5000e-07
[task 2024-01-11T21:51:00.988Z] [2024-01-11 21:51:00] [Trainer] [INFO] Reading original for epoch 104
[task 2024-01-11T21:51:01.215Z] [2024-01-11 21:51:01] Ep. 1 : Up. 25 : Sen. 49,999 : Cost 0.40901086 : Time 0.34s : 785027.10 words/s : gNorm 29.7880 : L.r. 4.6875e-07
[task 2024-01-11T21:51:01.216Z] [2024-01-11 21:51:01] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz
[task 2024-01-11T21:51:01.486Z] [2024-01-11 21:51:01] Saving Adam parameters
[task 2024-01-11T21:51:01.879Z] [2024-01-11 21:51:01] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz and /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.optimizer.npz
[task 2024-01-11T21:51:02.670Z] [2024-01-11 21:51:02] Ep. 1 : Up. 26 : Sen. 51,303 : Cost 0.32463440 : Time 1.45s : 168525.26 words/s : gNorm 29.5974 : L.r. 4.8750e-07
[task 2024-01-11T21:51:03.004Z] [2024-01-11 21:51:03] Ep. 1 : Up. 27 : Sen. 52,823 : Cost 0.27760595 : Time 0.33s : 245952.55 words/s : gNorm 29.2367 : L.r. 5.0625e-07
[task 2024-01-11T21:51:03.282Z] [2024-01-11 21:51:03] Ep. 1 : Up. 28 : Sen. 56,239 : Cost 0.65265727 : Time 0.28s : 73532.77 words/s : gNorm 29.0209 : L.r. 5.2500e-07
[task 2024-01-11T21:51:03.491Z] [2024-01-11 21:51:03] Ep. 1 : Up. 29 : Sen. 58,813 : Cost 0.73784047 : Time 0.21s : 136118.08 words/s : gNorm 28.2502 : L.r. 5.4375e-07
[task 2024-01-11T21:51:03.700Z] [2024-01-11 21:51:03] Ep. 1 : Up. 30 : Sen. 60,029 : Cost 0.55778843 : Time 0.21s : 266290.94 words/s : gNorm 28.9339 : L.r. 5.6250e-07
[task 2024-01-11T21:51:03.756Z] [2024-01-11 21:51:03] [Trainer] [INFO] Reading original for epoch 105
[task 2024-01-11T21:51:04.141Z] [2024-01-11 21:51:04] Ep. 1 : Up. 31 : Sen. 61,677 : Cost 0.24544582 : Time 0.44s : 436827.33 words/s : gNorm 28.7577 : L.r. 5.8125e-07
[task 2024-01-11T21:51:04.382Z] [2024-01-11 21:51:04] Ep. 1 : Up. 32 : Sen. 62,893 : Cost 0.33286783 : Time 0.24s : 151576.70 words/s : gNorm 28.5925 : L.r. 6.0000e-07
[task 2024-01-11T21:51:04.627Z] [2024-01-11 21:51:04] Ep. 1 : Up. 33 : Sen. 64,109 : Cost 0.31747290 : Time 0.25s : 178223.74 words/s : gNorm 28.4482 : L.r. 6.1875e-07
[task 2024-01-11T21:51:04.998Z] [2024-01-11 21:51:04] Ep. 1 : Up. 34 : Sen. 65,629 : Cost 0.34261644 : Time 0.37s : 672403.87 words/s : gNorm 28.3888 : L.r. 6.3750e-07
[task 2024-01-11T21:51:05.382Z] [2024-01-11 21:51:05] Ep. 1 : Up. 35 : Sen. 67,597 : Cost 0.35571006 : Time 0.38s : 718827.29 words/s : gNorm 28.6024 : L.r. 6.5625e-07
[task 2024-01-11T21:51:05.766Z] [2024-01-11 21:51:05] Ep. 1 : Up. 36 : Sen. 68,813 : Cost 0.36565000 : Time 0.38s : 614567.84 words/s : gNorm 28.5208 : L.r. 6.7500e-07
[task 2024-01-11T21:51:06.202Z] [2024-01-11 21:51:06] Ep. 1 : Up. 37 : Sen. 70,981 : Cost 0.31668627 : Time 0.44s : 601398.77 words/s : gNorm 28.4712 : L.r. 6.9375e-07
[task 2024-01-11T21:51:06.505Z] [2024-01-11 21:51:06] [Trainer] [INFO] Reading original for epoch 106
[task 2024-01-11T21:51:06.527Z] [2024-01-11 21:51:06] Ep. 1 : Up. 38 : Sen. 79,613 : Cost 1.45151091 : Time 0.32s : 79763.76 words/s : gNorm 28.3254 : L.r. 7.1250e-07
[task 2024-01-11T21:51:06.805Z] [2024-01-11 21:51:06] Ep. 1 : Up. 39 : Sen. 83,501 : Cost 0.79896307 : Time 0.28s : 41936.29 words/s : gNorm 28.2971 : L.r. 7.3125e-07
[task 2024-01-11T21:51:07.229Z] [2024-01-11 21:51:07] Ep. 1 : Up. 40 : Sen. 84,909 : Cost 0.24403062 : Time 0.42s : 402120.50 words/s : gNorm 28.1050 : L.r. 7.5000e-07
[task 2024-01-11T21:51:07.748Z] [2024-01-11 21:51:07] Ep. 1 : Up. 41 : Sen. 86,557 : Cost 0.27117914 : Time 0.52s : 483484.91 words/s : gNorm 28.0122 : L.r. 7.6875e-07
[task 2024-01-11T21:51:08.015Z] [2024-01-11 21:51:08] Ep. 1 : Up. 42 : Sen. 88,949 : Cost 0.47187948 : Time 0.27s : 80514.67 words/s : gNorm 27.6742 : L.r. 7.8750e-07
[task 2024-01-11T21:51:08.234Z] [2024-01-11 21:51:08] Ep. 1 : Up. 43 : Sen. 91,458 : Cost 0.78953999 : Time 0.22s : 767784.12 words/s : gNorm 28.4257 : L.r. 8.0625e-07
[task 2024-01-11T21:51:08.720Z] [2024-01-11 21:51:08] Ep. 1 : Up. 44 : Sen. 92,978 : Cost 0.27205503 : Time 0.49s : 515116.38 words/s : gNorm 28.3655 : L.r. 8.2500e-07
[task 2024-01-11T21:51:09.171Z] [2024-01-11 21:51:09] Ep. 1 : Up. 45 : Sen. 94,498 : Cost 0.31635004 : Time 0.45s : 508274.66 words/s : gNorm 28.5933 : L.r. 8.4375e-07
[task 2024-01-11T21:51:09.328Z] [2024-01-11 21:51:09] [Trainer] [INFO] Reading original for epoch 107
[task 2024-01-11T21:51:09.532Z] [2024-01-11 21:51:09] Ep. 1 : Up. 46 : Sen. 96,146 : Cost 0.35389695 : Time 0.36s : 437384.30 words/s : gNorm 28.8467 : L.r. 8.6250e-07
[task 2024-01-11T21:51:09.739Z] [2024-01-11 21:51:09] Ep. 1 : Up. 47 : Sen. 97,362 : Cost 0.34915510 : Time 0.21s : 98049.80 words/s : gNorm 28.6329 : L.r. 8.8125e-07
[task 2024-01-11T21:51:10.230Z] [2024-01-11 21:51:10] Ep. 1 : Up. 48 : Sen. 98,578 : Cost 0.26961991 : Time 0.49s : 470376.60 words/s : gNorm 28.8304 : L.r. 9.0000e-07
[task 2024-01-11T21:51:10.589Z] [2024-01-11 21:51:10] Ep. 1 : Up. 49 : Sen. 101,994 : Cost 0.62859082 : Time 0.36s : 590204.53 words/s : gNorm 29.1685 : L.r. 9.1875e-07
[task 2024-01-11T21:51:10.839Z] [2024-01-11 21:51:10] Ep. 1 : Up. 50 : Sen. 103,514 : Cost 0.37832299 : Time 0.25s : 111843.74 words/s : gNorm 28.7747 : L.r. 9.3750e-07
[task 2024-01-11T21:51:10.840Z] [2024-01-11 21:51:10] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz
[task 2024-01-11T21:51:11.224Z] [2024-01-11 21:51:11] Saving Adam parameters
[task 2024-01-11T21:51:11.609Z] [2024-01-11 21:51:11] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz and /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.optimizer.npz
[task 2024-01-11T21:51:12.396Z] [2024-01-11 21:51:12] [Trainer] [INFO] Reading original for epoch 108
[task 2024-01-11T21:51:13.074Z] [2024-01-11 21:51:13] Training finished
[task 2024-01-11T21:51:13.589Z] [2024-01-11 21:51:13] [valid] First sentence's tokens as scored:
[task 2024-01-11T21:51:13.590Z] [2024-01-11 21:51:13] [valid] Decoding validation set with SentencePieceVocab for scoring
[task 2024-01-11T21:51:13.590Z] [2024-01-11 21:51:13] [valid]   Hyp: С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С С
[task 2024-01-11T21:51:13.591Z] [2024-01-11 21:51:13] [valid]   Ref: T h e T V N e t w o r k R e p o r t s T h a t U n d e r T h e A g r e e m e n t R e a c h e d , M u s k M u s t L e a v e H i s P o s t A s C h a i r m a n O f T e s l a ’ s B o a r d O f D i r e c t o r s .
[task 2024-01-11T21:51:15.340Z] [2024-01-11 21:51:15] [Trainer] [INFO] Reading original for epoch 109
[task 2024-01-11T21:51:18.234Z] [2024-01-11 21:51:18] [Trainer] [INFO] Reading original for epoch 110
[task 2024-01-11T21:51:21.124Z] [2024-01-11 21:51:21] [Trainer] [INFO] Reading original for epoch 111
[task 2024-01-11T21:51:21.724Z] [2024-01-11 21:51:21] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-chrf.npz
[task 2024-01-11T21:51:21.868Z] [2024-01-11 21:51:21] [valid] Ep. 1 : Up. 50 : chrf : 0.0108796 : new best
[task 2024-01-11T21:51:22.179Z] [2024-01-11 21:51:22] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-ce-mean-words.npz
[task 2024-01-11T21:51:22.329Z] [2024-01-11 21:51:22] [valid] Ep. 1 : Up. 50 : ce-mean-words : 7.47263 : new best
[task 2024-01-11T21:51:24.036Z] [2024-01-11 21:51:24] [Trainer] [INFO] Reading original for epoch 112
[task 2024-01-11T21:51:26.881Z] [2024-01-11 21:51:26] [Trainer] [INFO] Reading original for epoch 113
[task 2024-01-11T21:51:29.718Z] [2024-01-11 21:51:29] [Trainer] [INFO] Reading original for epoch 114
[task 2024-01-11T21:51:29.750Z] [2024-01-11 21:51:29] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-bleu-detok.npz
[task 2024-01-11T21:51:29.900Z] [2024-01-11 21:51:29] [valid] Ep. 1 : Up. 50 : bleu-detok : 0 : new best
[task 2024-01-11T21:51:29.900Z] [2024-01-11 21:51:29] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz
[task 2024-01-11T21:51:30.216Z] [2024-01-11 21:51:30] Saving Adam parameters
[task 2024-01-11T21:51:30.594Z] [2024-01-11 21:51:30] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz and /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.optimizer.npz
[task 2024-01-11T21:51:32.532Z] [2024-01-11 21:51:32] [Trainer] [INFO] Reading original for epoch 115
[task 2024-01-11T21:51:35.376Z] [2024-01-11 21:51:35] [Trainer] [INFO] Reading original for epoch 116
[task 2024-01-11T21:51:38.167Z] [2024-01-11 21:51:38] [Trainer] [INFO] Reading original for epoch 117
[task 2024-01-11T21:51:40.962Z] [2024-01-11 21:51:40] [Trainer] [INFO] Reading original for epoch 118
[task 2024-01-11T21:51:43.752Z] [2024-01-11 21:51:43] [Trainer] [INFO] Reading original for epoch 119
[task 2024-01-11T21:51:46.943Z] [2024-01-11 21:51:46] [Trainer] [INFO] Reading original for epoch 120
[task 2024-01-11T21:51:50.085Z] [2024-01-11 21:51:50] [Trainer] [INFO] Reading original for epoch 121
[task 2024-01-11T21:51:53.263Z] [2024-01-11 21:51:53] [Trainer] [INFO] Reading original for epoch 122
[task 2024-01-11T21:51:56.415Z] [2024-01-11 21:51:56] [Trainer] [INFO] Reading original for epoch 123
[task 2024-01-11T21:51:59.506Z] [2024-01-11 21:51:59] [Trainer] [INFO] Reading original for epoch 124
[task 2024-01-11T21:52:02.635Z] [2024-01-11 21:52:02] [Trainer] [INFO] Reading original for epoch 125
[task 2024-01-11T21:52:05.737Z] [2024-01-11 21:52:05] [Trainer] [INFO] Reading original for epoch 126
[task 2024-01-11T21:52:08.843Z] [2024-01-11 21:52:08] [Trainer] [INFO] Reading original for epoch 127
[task 2024-01-11T21:52:11.912Z] [2024-01-11 21:52:11] [Trainer] [INFO] Reading original for epoch 128
[task 2024-01-11T21:52:14.901Z] [2024-01-11 21:52:14] [Trainer] [INFO] Reading original for epoch 129
[task 2024-01-11T21:52:17.764Z] [2024-01-11 21:52:17] [Trainer] [INFO] Reading original for epoch 130
[task 2024-01-11T21:52:20.608Z] [2024-01-11 21:52:20] [Trainer] [INFO] Reading original for epoch 131
[task 2024-01-11T21:52:23.441Z] [2024-01-11 21:52:23] [Trainer] [INFO] Reading original for epoch 132
[task 2024-01-11T21:52:26.258Z] [2024-01-11 21:52:26] [Trainer] [INFO] Reading original for epoch 133
[task 2024-01-11T21:52:29.062Z] [2024-01-11 21:52:29] [Trainer] [INFO] Reading original for epoch 134
[task 2024-01-11T21:52:31.893Z] [2024-01-11 21:52:31] [Trainer] [INFO] Reading original for epoch 135
[task 2024-01-11T21:52:34.754Z] [2024-01-11 21:52:34] [Trainer] [INFO] Reading original for epoch 136
[task 2024-01-11T21:52:37.589Z] [2024-01-11 21:52:37] [Trainer] [INFO] Reading original for epoch 137
[task 2024-01-11T21:52:40.411Z] [2024-01-11 21:52:40] [Trainer] [INFO] Reading original for epoch 138
[task 2024-01-11T21:52:43.247Z] [2024-01-11 21:52:43] [Trainer] [INFO] Reading original for epoch 139
[task 2024-01-11T21:52:46.072Z] [2024-01-11 21:52:46] [Trainer] [INFO] Reading original for epoch 140
[task 2024-01-11T21:52:48.883Z] [2024-01-11 21:52:48] [Trainer] [INFO] Reading original for epoch 141
[task 2024-01-11T21:52:51.727Z] [2024-01-11 21:52:51] [Trainer] [INFO] Reading original for epoch 142
[task 2024-01-11T21:52:54.566Z] [2024-01-11 21:52:54] [Trainer] [INFO] Reading original for epoch 143
[task 2024-01-11T21:52:57.378Z] [2024-01-11 21:52:57] [Trainer] [INFO] Reading original for epoch 144
[task 2024-01-11T21:53:01.122Z] [2024-01-11 21:53:01] [Trainer] [INFO] Reading original for epoch 145
[task 2024-01-11T21:53:04.551Z] [2024-01-11 21:53:04] [Trainer] [INFO] Reading original for epoch 146
[task 2024-01-11T21:53:08.024Z] [2024-01-11 21:53:08] [Trainer] [INFO] Reading original for epoch 147
[task 2024-01-11T21:53:11.517Z] [2024-01-11 21:53:11] [Trainer] [INFO] Reading original for epoch 148
[task 2024-01-11T21:53:14.987Z] [2024-01-11 21:53:14] [Trainer] [INFO] Reading original for epoch 149
[task 2024-01-11T21:53:18.439Z] [2024-01-11 21:53:18] [Trainer] [INFO] Reading original for epoch 150
[task 2024-01-11T21:53:21.827Z] [2024-01-11 21:53:21] [Trainer] [INFO] Reading original for epoch 151
[task 2024-01-11T21:53:25.192Z] [2024-01-11 21:53:25] [Trainer] [INFO] Reading original for epoch 152
[task 2024-01-11T21:53:28.544Z] [2024-01-11 21:53:28] [Trainer] [INFO] Reading original for epoch 153
[task 2024-01-11T21:53:31.883Z] [2024-01-11 21:53:31] [Trainer] [INFO] Reading original for epoch 154
[task 2024-01-11T21:53:35.250Z] [2024-01-11 21:53:35] [Trainer] [INFO] Reading original for epoch 155
[task 2024-01-11T21:53:38.568Z] [2024-01-11 21:53:38] [Trainer] [INFO] Reading original for epoch 156
[task 2024-01-11T21:53:41.879Z] [2024-01-11 21:53:41] [Trainer] [INFO] Reading original for epoch 157
[task 2024-01-11T21:53:45.225Z] [2024-01-11 21:53:45] [Trainer] [INFO] Reading original for epoch 158
[task 2024-01-11T21:53:48.541Z] [2024-01-11 21:53:48] [Trainer] [INFO] Reading original for epoch 159
[task 2024-01-11T21:53:51.772Z] [2024-01-11 21:53:51] [Trainer] [INFO] Reading original for epoch 160
[task 2024-01-11T21:53:54.896Z] [2024-01-11 21:53:54] [Trainer] [INFO] Reading original for epoch 161
[task 2024-01-11T21:53:57.889Z] [2024-01-11 21:53:57] [Trainer] [INFO] Reading original for epoch 162
[task 2024-01-11T21:54:00.806Z] [2024-01-11 21:54:00] [Trainer] [INFO] Reading original for epoch 163
[task 2024-01-11T21:54:03.720Z] [2024-01-11 21:54:03] [Trainer] [INFO] Reading original for epoch 164
[task 2024-01-11T21:54:06.552Z] [2024-01-11 21:54:06] [Trainer] [INFO] Reading original for epoch 165
[task 2024-01-11T21:54:09.398Z] [2024-01-11 21:54:09] [Trainer] [INFO] Reading original for epoch 166
[task 2024-01-11T21:54:12.271Z] [2024-01-11 21:54:12] [Trainer] [INFO] Reading original for epoch 167
[task 2024-01-11T21:54:15.120Z] [2024-01-11 21:54:15] [Trainer] [INFO] Reading original for epoch 168
[task 2024-01-11T21:54:17.971Z] [2024-01-11 21:54:17] [Trainer] [INFO] Reading original for epoch 169
[task 2024-01-11T21:54:20.803Z] [2024-01-11 21:54:20] [Trainer] [INFO] Reading original for epoch 170
[task 2024-01-11T21:54:23.680Z] [2024-01-11 21:54:23] [Trainer] [INFO] Reading original for epoch 171
[task 2024-01-11T21:54:26.530Z] [2024-01-11 21:54:26] [Trainer] [INFO] Reading original for epoch 172
[task 2024-01-11T21:54:29.397Z] [2024-01-11 21:54:29] [Trainer] [INFO] Reading original for epoch 173
[task 2024-01-11T21:54:32.279Z] [2024-01-11 21:54:32] [Trainer] [INFO] Reading original for epoch 174
[task 2024-01-11T21:54:35.139Z] [2024-01-11 21:54:35] [Trainer] [INFO] Reading original for epoch 175
[task 2024-01-11T21:54:38.010Z] [2024-01-11 21:54:38] [Trainer] [INFO] Reading original for epoch 176
[task 2024-01-11T21:54:40.855Z] [2024-01-11 21:54:40] [Trainer] [INFO] Reading original for epoch 177
[task 2024-01-11T21:54:43.702Z] [2024-01-11 21:54:43] [Trainer] [INFO] Reading original for epoch 178
[task 2024-01-11T21:54:46.561Z] [2024-01-11 21:54:46] [Trainer] [INFO] Reading original for epoch 179
[task 2024-01-11T21:54:49.409Z] [2024-01-11 21:54:49] [Trainer] [INFO] Reading original for epoch 180
[task 2024-01-11T21:54:52.274Z] [2024-01-11 21:54:52] [Trainer] [INFO] Reading original for epoch 181
[task 2024-01-11T21:54:55.127Z] [2024-01-11 21:54:55] [Trainer] [INFO] Reading original for epoch 182
[task 2024-01-11T21:54:58.070Z] [2024-01-11 21:54:58] [Trainer] [INFO] Reading original for epoch 183
[task 2024-01-11T21:55:00.937Z] [2024-01-11 21:55:00] [Trainer] [INFO] Reading original for epoch 184
[task 2024-01-11T21:55:03.822Z] [2024-01-11 21:55:03] [Trainer] [INFO] Reading original for epoch 185
[task 2024-01-11T21:55:06.677Z] [2024-01-11 21:55:06] [Trainer] [INFO] Reading original for epoch 186
[task 2024-01-11T21:55:09.542Z] [2024-01-11 21:55:09] [Trainer] [INFO] Reading original for epoch 187
[task 2024-01-11T21:55:12.444Z] [2024-01-11 21:55:12] [Trainer] [INFO] Reading original for epoch 188
[task 2024-01-11T21:55:15.354Z] [2024-01-11 21:55:15] [Trainer] [INFO] Reading original for epoch 189
[task 2024-01-11T21:55:59.336Z] [2024-01-11 21:55:59] [Trainer] [INFO] trainer stopped reading input
[task 2024-01-11T21:55:59.429Z] + cp /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-chrf.npz /home/ubuntu/tasks/task_170500930891517/artifacts/final.model.npz.best-chrf.npz
[task 2024-01-11T21:55:59.464Z] + cp /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-chrf.npz.decoder.yml /home/ubuntu/tasks/task_170500930891517/artifacts/final.model.npz.best-chrf.npz.decoder.yml
[task 2024-01-11T21:55:59.467Z] + echo '### Model training is completed: /home/ubuntu/tasks/task_170500930891517/artifacts'
[task 2024-01-11T21:55:59.467Z] ### Model training is completed: /home/ubuntu/tasks/task_170500930891517/artifacts
[task 2024-01-11T21:55:59.467Z] + echo '###### Done: Training a model'
[task 2024-01-11T21:55:59.467Z] ###### Done: Training a model
[task 2024-01-11T21:55:59.467Z] + '[' None == None ']'
[task 2024-01-11T21:55:59.467Z] + cp /home/ubuntu/tasks/task_170500930891517/fetches/vocab.spm /home/ubuntu/tasks/task_170500930891517/artifacts
[fetches 2024-01-11T21:55:59.470Z] removing /home/ubuntu/tasks/task_170500930891517/fetches
[fetches 2024-01-11T21:55:59.820Z] finished
[taskcluster 2024-01-11T21:55:59.832Z]    Exit Code: 0
[taskcluster 2024-01-11T21:55:59.832Z]    User Time: 14m15.61685s
[taskcluster 2024-01-11T21:55:59.832Z]  Kernel Time: 1m42.534363s
[taskcluster 2024-01-11T21:55:59.832Z]    Wall Time: 11m49.075366593s
[taskcluster 2024-01-11T21:55:59.832Z]       Result: SUCCEEDED
[taskcluster 2024-01-11T21:55:59.832Z] === Task Finished ===
[taskcluster 2024-01-11T21:55:59.832Z] Task Duration: 11m49.077502565s
[taskcluster 2024-01-11T21:56:02.333Z] Uploading artifact public/build/config.opustrainer.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/config.opustrainer.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:02.528Z] Uploading artifact public/build/config.opustrainer.yml.state from file /home/ubuntu/tasks/task_170500930891517/artifacts/config.opustrainer.yml.state with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:02.705Z] Uploading artifact public/build/devset.out from file /home/ubuntu/tasks/task_170500930891517/artifacts/devset.out with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:02.896Z] Uploading artifact public/build/final.model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/final.model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:04.878Z] Uploading artifact public/build/final.model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/final.model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:05.068Z] Uploading artifact public/build/model.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:06.966Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-bleu-detok.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:08.882Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz.decoder.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-bleu-detok.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:09.068Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-ce-mean-words.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:10.987Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz.decoder.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-ce-mean-words.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:11.189Z] Uploading artifact public/build/model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:13.161Z] Uploading artifact public/build/model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:13.337Z] Uploading artifact public/build/model.npz.decoder.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:13.508Z] Uploading artifact public/build/model.npz.optimizer.npz from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.optimizer.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:20.854Z] Uploading artifact public/build/model.npz.progress.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.progress.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:21.037Z] Uploading artifact public/build/model.npz.yml from file /home/ubuntu/tasks/task_170500930891517/artifacts/model.npz.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:21.234Z] Uploading artifact public/build/opustrainer.log from file /home/ubuntu/tasks/task_170500930891517/artifacts/opustrainer.log with content encoding "gzip", mime type "text/plain" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:21.572Z] Uploading artifact public/build/train.log from file /home/ubuntu/tasks/task_170500930891517/artifacts/train.log with content encoding "gzip", mime type "text/plain" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:21.765Z] Uploading artifact public/build/valid.log from file /home/ubuntu/tasks/task_170500930891517/artifacts/valid.log with content encoding "gzip", mime type "text/plain" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:21.948Z] Uploading artifact public/build/vocab.spm from file /home/ubuntu/tasks/task_170500930891517/artifacts/vocab.spm with content encoding "gzip", mime type "application/x-source-rpm" and expiry 2024-04-10T20:54:29.096Z
[taskcluster 2024-01-11T21:56:22.093Z] [mounts] Preserving cache: Moving "/home/ubuntu/tasks/task_170500930891517/checkouts" to "/home/ubuntu/caches/XfqD9z7bQh-bB0jGABrskw"
[taskcluster 2024-01-11T21:56:22.152Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-04-10T20:54:29.096Z