forked from apache/tez
-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathCHANGES.txt
2500 lines (2415 loc) · 170 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Apache Tez Change Log
=====================
Release 0.9.2: 2019-03-18
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4052. Fit dot files ASF License issues - part 2 (Jonathan Eagles via kshukla)
TEZ-4044. Zookeeper: exclude jline from Zookeeper client from tez dist
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (addendum)
TEZ-4031. Support tez gitbox migration (Jonathan Eagles via kshukla)
TEZ-4048. Make proto history logger queue size configurable
TEZ-4028: Events not visible from proto history logging for s3a filesystem until dag completes (Harish JP, via Gopal V)
TEZ-4047. Tez trademark in xml is causing xml parsing issue (Jonathan Eagles via kshukla)
TEZ-4050. maven site is failing due to missing configuration. (Jonathan Eagles via kshukla)
TEZ-4049. Fix findbugs issues in NotRunningJob (Jonathan Eagles via kshukla)
TEZ-4032. TEZ will throw Client cannot authenticate via:[TOKEN, KERBEROS] when used with HDFS federation(non viewfs, only hdfs schema used).
TEZ-4042. Speculative attempts should avoid running on the same node
TEZ-4035. Tez master breaks with YARN 3.2.0 ApplicationReport API change (jeagles)
TEZ-3952. Allow Tez task speculation to grant greater customization of certain parameters (Nishant Dash via jeagles)
TEZ-4034. Column selector filter should be case-insensitive (Jacob Tolar via jeagles)
TEZ-4043. Create a yetus compatible checkstyle configuration (Jonathan Eagles via kshukla)
TEZ-4004. Update jetty9 to align with Hadoop and Hive (Jonathan Eagles via kshukla)
TEZ-4041. TestExtServicesWithLocalMode fails in docker (Jonathan Eagles via kshukla)
TEZ-4040. Upgrade RoaringBitmap version to avoid NoSuchMethodError (Jonathan Eagles via kshukla)
TEZ-4037. Add back DAG search status KILLED (Jonathan Eagles via kshukla)
TEZ-4036. TestMockDAGAppMaster#testInternalPreemption should assert for failed state (Kuhu Shukla via jeagles)
TEZ-4027. DagAwareYarnTaskScheduler can miscompute blocked vertices and cause a hang
TEZ-4022. Upgrade Maven Surefire plugin to 3.0.0-M1
TEZ-4012. Add docker support for Tez. (Jonathan Eagles via kshukla)
TEZ-3961. Tez UI web.xml tries to reach out to java.sun.com for validation after moving to jetty-9 (Kuhu Shukla via jeagles)
TEZ-3995. Addendum (Jaume Marhuenda via jegales)
TEZ-3990. The number of shuffle penalties for a host/inputAttemptIdentifier should be capped (Kuhu Shukla via jeagles)
TEZ-4003. Add [email protected] to KEYS file (Gopal V via jeagles)
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (Jaume Marhuenda via jegales)
TEZ-3969. TaskAttemptImpl: static fields initialized in instance ctor (Jaume Marhuenda via jegales)
TEZ-3994. Upgrade maven-surefire-plugin to 0.21.0 to support yetus
TEZ-3975. Add OWASP Dependency Check to the build
TEZ-3981. UnorderedPartitionedKVWriter.getInitialMemoryRequirement may return negative memory (Jaume M via jeagles)
TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jlowe)
Revert "TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jeagles)"
TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jeagles)
TEZ-3972. Tez DAG can hang when a single task fails to fetch (Kuhu Shukla via jeagles)
TEZ-3973. Add Kuhu Shukla's (kshukla) public key to KEYS (Kuhu Shukla via jeagles)
TEZ-3980: ShuffleRunner: the wake loop needs to check for shutdown (Gopal V, reviewed by Gunther Hagleitner)
TEZ-3977. Add Eric Wohlstadter's public key to KEYS (Eric Wohlstadter via jeagles)
TEZ-3978. DAGClientServer Socket exception when localhost name lookup failures (Jonathan Eagles via jlowe)
TEZ-3974: Correctness regression of TEZ-955 in TEZ-2937 (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3934. LegacySpeculator sometime issues wrong number of speculative attempts (Nishant Dash via jeagles)
TEZ-3942. RPC getTask writable optimization invalid in hadoop 2.8+
TEZ-3970. NullPointerException in Tez ShuffleHandler Ranged Fetch (Jonathan Eagles via kshukla)
TEZ-3964. Inflater not closed in some places (Jaume M via jlowe)
TEZ-3912. Fetchers should be more robust to corrupted inputs (Kuhu Shukla via jeagles)
TEZ-3963. Possible InflaterInputStream leaked in TezCommonUtils and related classes (Jaume M via jlowe)
TEZ-3954. Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords (Jonathan Eagles via kshukla)
TEZ-3953: Restore ABI-compat for DAGClient for TEZ-3951 (Sergey Shelukhin via Gopal V)
TEZ-3960. Better error handling in proto history logger and add doAs support. (harishjp)
TEZ-3962. Configuration decode leaks an Inflater object (Eric Wohlstadter via jlowe)
TEZ-3959. HTTP 502 for bower install (Harish Jaiprakash via Sree)
TEZ-3951. TezClient wait too long for the DAGClient for prewarm; tries to shut down the wrong DAG (Sergey Shelukhin via Harish Jaiprakash)
TEZ-3938. Task attempts failing due to not making progress (Kuhu Shukla via jeagles)
TEZ-3947. TestATSHistoryWithACLs fails with Hadoop3 and Jersey 1.19 (Eric Wohlstadter via jeagles)
TEZ-3943. TezClient leaks DAGClient for prewarm (Sergey Shelukhin via jlowe)
TEZ-3939. Remove performance hit of precondition check in AM for register running task attempt (Jonathan Eagles via jlowe)
TEZ-3940. Reduce time to convert TaskFinishedEvent to string (Jonathan Eagles via jlowe)
TEZ-3935. DAG aware scheduler should release unassigned new containers rather than hold them (Jason Lowe via jeagles)
TEZ-3937. Empty partition BitSet to byte[] conversion creates one extra byte in rounding error (Jonathan Eagles via jlowe)
TEZ-3824. MRCombiner creates new JobConf copy per spill (Jonathan Eagles via jlowe)
TEZ-3933. Remove sleep from test TestExceptionPropagation (Jonathan Eagles via kshukla)
TEZ-3932. TaskSchedulerManager can throw NullPointerException during DAGAppMaster container cleanup race (Jonathan Eagles via jlowe)
TEZ-3931. TestExternalTezServices fails on Hadoop3 (Jonathan Eagles via kshukla)
TEZ-3930. TestDagAwareYarnTaskScheduler fails on Hadoop 3 (Jason Lowe via jeagles)
TEZ-3927. TestReduceProcessor fails on Hadoop 3.x (Jonathan Eagles via kshukla)
TEZ-3924. TestDefaultSorter fails intermittently due random keys and interaction with RLE and partition collisions (Jonathan Eagles via kshukla)
TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)
Revert "TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)"
TEZ-3873. A maven enforcer plugin dependency error in pom.xml (Jinjiang Ling via jeagles)
TEZ-3887. Tez Shuffle Handler should support Index Cache configuration (Jonathan Eagles via kshukla)
TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)
TEZ-3817. DAGs can hang after more than one uncaught Exception during doTransition. (kshukla)
TEZ-3915. Create protobuf based history event logger. (Harish Jaiprakash, reviewed by Gunther Hagleitner)
TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)
Revert "TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)"
TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)
Revert "TEZ-3902. Upgrade to netty-3.10.5.Final.jar (Jason Lowe via kshukla)"
TEZ-3902. Upgrade to netty-3.10.5.Final.jar (Jason Lowe via kshukla)
TEZ-3909. DAG can hang if vertex with no tasks is killed (Jason Lowe via jeagles)
TEZ-3907. Improve log message to include the location the writers decide to spill output (Kuhu Shukla via jlowe)
TEZ-3905: Change BUILDING.TXT to minimum JDK 1.8 (Eric Wohlstadter via Gopal V)
TEZ-3874. NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration. (Eric Wohlstadter via jlowe)
TEZ-3892: getClient API for TezClient (Eric Wohlstadter via Gopal V)
TEZ-3897. Tez Local Mode hang for vertices with broadcast input. (Jonathan Eagles via jlowe)
TEZ-3888: Update Jetty to org.eclipse.jetty 9.x (Eric Wohlstadter, reviewed by Rohini Palaniswamy)
TEZ-3898. TestTezCommonUtils fails when compiled against hadoop version >= 2.8 (Jason Lowe via jeagles)
TEZ-3896. TestATSV15HistoryLoggingService#testNonSessionDomains is failing (Jason Lowe via jeagles)
TEZ-3893. Tez Local Mode can hang for cases. (Jonathan Eagles via jlowe)
TEZ-3894. Tez intermediate outputs implicitly rely on permissive umask for shuffle (Jason Lowe via kshukla)
TEZ-3895. Missing name for local mode task scheduler service async request handler thread (Jonathan Eagles via kshukla)
TEZ-3770. DAG-aware YARN task scheduler (jlowe)
TEZ-3880: Do not count rejected tasks as killed in vertex progress (Sergey Shelukhin, reviewed by Gunther Hagleitner)
TEZ-3877. Delete unordered spill files once merge is done (Jason Lowe via jeagles)
TEZ-3883. Update version in master to 0.9.2 (zhiyuany)
TEZ-3882. Changes for 0.9.1 release (zhiyuany)
Release 0.9.1: 2017-11-16
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3876. Bug in local mode distributed cache files (Jacob Tolar via jeagles)
TEZ-3869. Analyzer: Fix VertexInfo::getLastTaskToFinish comparison
TEZ-3868. Update website to factor in the TEZ trademark registration
TEZ-3867. testSendCustomProcessorEvent try to get array out of read only ByteBuffer
TEZ-3855. Allow vertex manager to send event to processor
TEZ-3861. PipelineSorter setting negative progess
TEZ-3857. Tez TaskImpl can throw Invalid state transition for leaf tasks that do Retro Active Transition
TEZ-3862. Tez UI: Upgrade em-tgraph to version 0.0.14
TEZ-3858. Misleading dag level diagnostics in case of invalid vertex event
TEZ-3666. Integer overflow in ShuffleVertexManagerBase
TEZ-3856. API to access counters in InputInitializerContext
TEZ-3849. Combiner+PipelinedSorter silently drops records
TEZ-3853. Binary incompatibility caused by DEFAULT_LOG_LEVEL
TEZ-3854. Make use of new improved em-table sort-icon
TEZ-3850. Enable header as sort button on Tez UI
TEZ-3852. Optimize ContainerContext.isSuperSet to speed container reuse decisions
TEZ-3844. Tez UI Dag Counters show no records for a RUNNING DAG
TEZ-3848. Tez Local mode doesn't localize distributed cache files
TEZ-3847. AM web controller task counters are empty sometimes
TEZ-3830. HistoryEventTimelineConversion should not hard code the Task state
TEZ-3833. Tasks should report codec errors during shuffle as fetch failures
TEZ-3845. Tez UI Cleanup Stats Table
TEZ-3843. Tez UI Vertex/Tasks log links for running tasks are missing
TEZ-3836. Tez UI task page sort does not work on RHEL7/Fedora
TEZ-3840 addendum. Tez should write TEZ_DAG_ID before TEZ_EXTRA_INFO
TEZ-3840. Tez should write TEZ_DAG_ID before TEZ_EXTRA_INFO
TEZ-3839. Tez Shuffle Handler prints disk error stack traces for every read failure.
TEZ-3834. TaskSchedulerManager NullPointerException during shutdown when failed to start
TEZ-3724. Tez UI on HTTP corrects HTTPS REST calls to HTTP
TEZ-3831 addendum. Reduce Unordered memory needed for storing empty completed events
TEZ-3832. TEZ DAG status shows SUCCEEDED for SUCCEEDED_WITH_FAILURES final status
TEZ-3831. Reduce Unordered memory needed for storing empty completed events
TEZ-3828. Allow relaxing locality when retried task's priority is kept same
TEZ-3827. TEZ Vertex status on DAG index page shows SUCCEEDED for SUCCEEDED_WITH_FAILURES final status
TEZ-3825. Tez UI DAGs page can't query RUNNING or SUBMITTED apps
TEZ-3431. Add unit tests for container release
TEZ-3816. Add ability to automatically speculate single-task vertices
TEZ-3813. Reduce Object size of MemoryFetchedInput for large jobs
TEZ-3803. Tasks can get killed due to insufficient progress while waiting for shuffle inputs to complete
TEZ-3804. FetcherOrderedGrouped#setupLocalDiskFetch should ignore empty partition records
TEZ-3807. InMemoryWriter is not tested with RLE enabled
TEZ-3212. IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
TEZ-3752. Reduce Object size of InMemoryMapOutput for large jobs
TEZ-3805. Analyzer: Add an analyzer to find out scheduling misses in 1:1 edges
TEZ-3797. Add tez debug tool for comparing counters of 2 DAGs
Release 0.9.0: 2017-07-21
INCOMPATIBLE CHANGES
TEZ-3693. ControlledClock is not used.
TEZ-3745. Change master to required java 8
TEZ-3689. Change minimum hadoop version to 2.7.0.
TEZ-3611. Create lightweight summary events for ATS.
TEZ-3652. Remove ShuffleClientMetrics
TEZ-3659. AM/Task classpath should not contain hadoop conf directory.
ALL CHANGES:
TEZ-3798. Remove duplicate package-info.java
TEZ-3792. RootInputVertexManager doesn't drain queued source task completed events
TEZ-3795. Vertex state machine can throw InvalidStateTransitonException from TERMINATING state
TEZ-3794. tez-tools: swimlane does not recognize HistoryEventHandler.criticalEvents based logs
TEZ-3791. Failed/Killed task can throw InvalidStateTransitonException when a new attempt is launched
TEZ-3787. Remove Tez UI build and rebuild errors and warning due to yarn install and ember-truth-helpers
TEZ-3786. Fix Tez UI test failures after TEZ-3775
TEZ-3784. Submitting very large DAG throws com.google.protobuf.CodedInputStream exception
TEZ-3274. Vertex with MRInput and broadcast input does not respect slow start
TEZ-3775. Tez UI: Show DAG context in document title
TEZ-3605. Detect and prune empty partitions for the Ordered case
TEZ-3771. Tez UI: WASB/ADLS counters should be listed on the Tez UI
TEZ-3769. Unordered: Fix wrong stats being sent out in the last event, when final merge is disabled
TEZ-3777. Avoid buffer copies by passing RLE flag to TezMerger from PipelinedSorter
TEZ-3767. Shuffle should not report error to AM during inputContext.killSelf()
TEZ-3778. Remove SecurityInfo from tez-auxservices shaded jar
TEZ-3762. When final merge is disabled in unordered case, it should create index file instead of relying on cache
TEZ-3761. NPE in Fetcher under load
TEZ-3758. Vertex can hang in RUNNING state when two task attempts finish very closely and have retroactive failures
TEZ-3768. Test timeout value for TestShuffleHandlerJobs is low
TEZ-3766. Tez Aux-services : Clean up shaded jar to not include default config xml files and yarn-client pieces
TEZ-3760. Tez AUX Services: Shading needs to filter SIG files with -Pazure builds
TEZ-3741. Tez outputs should free memory when closed
TEZ-3698: UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
TEZ-3732. Reduce Object size of InputAttemptIdentifier and MapOutput for large jobs
TEZ-3701. UnorderedPartitionedKVWriter - issues with parallel Deflater usage, synchronousqueue in threadpool
TEZ-3750. Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedPartitionedKVOutput
TEZ-3749. Get map and reduce task memory from JobConf
TEZ-3748. TaskAttemptImpl State Machine Invalid event: TA_SUBMITTED at KILL_IN_PROGRESS
TEZ-3739. Fair CartesianProduct doesn't works well with huge difference in output size
TEZ-3714. Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form Hive side
TEZ-3747. TezConstants.TEZ_SHUFFLE_HANDLER_SERVICE_ID is referenced in Hive
TEZ-3736. SubmittedDAGs is always 0 in Resource Manager UI.
TEZ-3744. Fix findbugs warnings after TEZ-3334 merge
TEZ-3743. TestTaskCommunicatorContextImpl throws NullPointerException after TEZ-3334 merge
TEZ-3742. Fix AMContainerHelpers#createCommonContainerLaunchContext to not pass localResources.
TEZ-3737. FairCartesianProductVertexMananger used incorrect #partition
TEZ-3691. Setup fetchers to use shared executor
TEZ-3662. Vertex Duration in 0.9 Tez UI regression from 0.7
TEZ-3730. Lower logging level in UnorderedPartitionedKVWriter.
TEZ-3723. TezIndexRecord#hasData() returns true for empty index record in the Unordered case
TEZ-3716. Allow attempt retries to be treated the same as the first attempt.
TEZ-3715. Differentiate between TaskAttempt submission and TaskAttempt started.
TEZ-3708. Improve parallelism and auto grouping of unpartitioned cartesian product
TEZ-3717. tez-yarn-timeline-history-with-fs does not build with hadoop-2.8.
TEZ-3697. Adding #output_record in vertex manager event payload
TEZ-3673. Allocate smaller buffers in UnorderedPartitionedKVWriter.
TEZ-3707. TezSharedExecutor race condition in awaitTermination vs isTerminated.
TEZ-3700. Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
TEZ-3703. Use a sha comparison to compare vertex and dag resources, if there is a mismatch.
TEZ-3699. For large dataset, pipelined shuffle throws exceptions in consumer side for UnorderedPartitioned edge
TEZ-3695. TestTezSharedExecutor fails sporadically.
TEZ-2049. Remove YARN references from Tez AsyncDispatcher
TEZ-3675. Handle changes to ResourceCalculatorProcessTree in YARN-3427 for Hadoop 3.x
TEZ-3690. Tez on hadoop 3 build failed due to hdfs client/server jar separation
TEZ-3687. Code smell in DAGStatus and VertexStatus equals implementation
TEZ-3631. Tez UI: TEZ_DAG_EXTRA_INFO compatibility changes - Makes All DAGs page faster
TEZ-3680. Optimizations to UnorderedPartitionedKVWriter
TEZ-1187. Add a framework ExecutorService which shares threads
TEZ-3654. Make CartesianProduct edge work with GroupInputEdge
TEZ-3285. Tez UI: Lock down dependency versions
TEZ-3683. LocalContainerLauncher#shouldDelete member variable is not used
TEZ-3681. Improve UI error message while trying to sort running DAGs with Auto Refresh enabled
TEZ-3668. Explicitly include hadoop-mapreduce-client-shuffle for test in root pom
TEZ-3667. Stop using org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
TEZ-3665. TestATSV15HistoryLoggingService should use mocked TimelineClient
TEZ-1526. LoadingCache for TezTaskID slow for large jobs
TEZ-3653. Tez UI: Swimlane tooltip is not proper for running DAGs
TEZ-3656. Tez UI: Status correction is not working as expected
TEZ-3650. Improve performance of FetchStatsLogger#logIndividualFetchComplete
TEZ-3655. Specify netty version instead of inheriting from hadoop dependency.
TEZ-3503. Tez UI: Support search by queue name
TEZ-3642. Tez UI: Auto-refresh is not stopping when DAG is the main entity
TEZ-3253. Remove special handling for last app attempt.
TEZ-3648. IFile.Write#close has an extra output stream flush
TEZ-3649. AsyncHttpConnection should add StopWatch start
TEZ-3647. Add a setting which lets Tez determine Xmx.
TEZ-3646. IFile.Writer has an extra output stream flush call
TEZ-3640. Tez UI: Add associated llap application id to queries page
TEZ-3639. Tez UI: Footer pagination is improper in landing page
TEZ-3637. TezMerger logs too much at INFO level.
TEZ-3638. VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
TEZ-3630. Tez UI: Use DAG status for controlling auto-refresh polling
TEZ-3634. reduce the default buffer sizes in PipelinedSorter by a small amount.
TEZ-3626. Tez UI: First Task Start Time & Last Task Finish Time values are showing up incorrectly
TEZ-3629. Tez UI: Enable the UI to display log links from LLAP
TEZ-3627. Use queue name available in RegisterApplicationMasterResponse for publishing to ATS.
TEZ-3610. TEZ UI 0.7 0.9 compatibility for url query params and tez-app sub-routes
TEZ-3619. Tez UI: Improve DAG Data download
TEZ-3615. Tez UI: Table changes
TEZ-3267. Publish queue name to ATS as part of dag summary.
TEZ-3602. Tez UI: Query Name field is not required
TEZ-3581. Add different logger to enable suppressing logs for specific lines.
TEZ-3600. Fix flaky test: TestTokenCache.
TEZ-3598. Tez UI: Text formatting changes
TEZ-3594. Tez UI: Graphical view tooltip issues
TEZ-3593. Tez UI: Issues in timeline page
TEZ-3592. Tez UI: Search issues
TEZ-3591. Tez UI: Logs url in all DAGs doesn't open in a new window
TEZ-3589. add a unit test for amKeepAlive not being shutdown if an app takes a long time to launch.
TEZ-3554. Add a link to get to all logs from Tez UI while job is running
TEZ-3417. Reduce sleep time on AM shutdown to reduce test runtimes
TEZ-3494. Support relative url for tez-ui.history-url.base config
TEZ-3575. RM have started forwarding origin. Use that in AMWebController for CORS support
TEZ-3580. Tez UI: Pagination broken on queries page
TEZ-3584. amKeepAliveService in TezClient should shutdown in case of AM failure.
TEZ-3583. Tez UI: UTs are flaky because of a dependency issue
TEZ-3579. Wrong configuration key for max slow start fraction in CartesianProductVertexManager.
TEZ-2712. Tez UI: Display the vertex description in the tooltip of vertex in DAG view UI
TEZ-3571. Tez UI: Display a Total Timeline View for Hive Queries
TEZ-3496. Tez UI: Optimize display of all tasks table
TEZ-3556. Tez UI: Display query configurations
TEZ-3531. Tez UI: All Queries table: Improve searchability
TEZ-3530. Tez UI: Add query details page, and link the page from All Queries table
TEZ-3529. Tez UI: Add 'All Queries' table in the landing page along 'All DAGs' page
TEZ-3458. Auto grouping for cartesian product edge(unpartitioned case).
TEZ-3443. Remove a repeated/unused method from MRTask.
TEZ-3551: FrameworkClient created twice causing minor delay
TEZ-3504. Tez UI: Duration is displaying invalid values when start or end time is invalid
TEZ-3570. Tez UI: Wait for sometime before tooltips are displayed
TEZ-3555. Tez UI: Build is failing in RHEL6
TEZ-3565: amConfig should check queuename isEmpty
TEZ-3558. CartesianProduct is missing from the ExampleDriver class
TEZ-3552. Shuffle split array when size-based sorting is turned off.
TEZ-3271. Provide mapreduce failures.maxpercent equivalent.
TEZ-3222. Reduce messaging overhead for auto-reduce parallelism case
TEZ-3547. Add TaskAssignment Analyzer
TEZ-3508. TestTaskScheduler cleanup.
TEZ-3269. Provide basic fair routing and scheduling functionality via custom VertexManager and EdgeManager.
TEZ-3477. MRInputHelpers generateInputSplitsToMem public API modified
TEZ-3465. Support broadcast edge into cartesian product vertex and forbid other edges.
TEZ-3502. Tez UI: Search in All DAGs page doesn't work with numeric values
TEZ-3470. Tez UI: Make the build work in IBM PPC
TEZ-3457. Add more unit test coverage for container reuse.
TEZ-3215. Support for MultipleOutputs.
TEZ-3484. Tez UI: Remove .travis.yml from webapp folder
TEZ-3405. Support ability for AM to kill itself if there is no client heartbeating to it.
TEZ-3469. Tez UI: Bump Phantom JS version to 2.1.1
TEZ-3430. Make split sorting optional.
TEZ-3466. Tez classpath building to mimic mapreduce classpath building
TEZ-3428. Tez UI: First Tab not needed for few entries in DAG listings
TEZ-3453. Correct the downloaded ATS dag data location for analyzer
TEZ-3449. Fix Spelling typos.
TEZ-3433. Tez UI: Searching using wrong ID causes error in all DAGs page
TEZ-3429. Set reconfigureDoneTime on VertexConfigurationDoneEvent properly.
TEZ-3163. Reuse and tune Inflaters and Deflaters to speed DME processing
TEZ-3434. Add unit tests for flushing of recovery events.
TEZ-3404. Move blocking call for YARN Timeline domain creation from client side to AM.
TEZ-3272. Add AMContainerImpl and AMNodeImpl to StateMachine visualization list.
TEZ-3284. Synchronization for every write in UnorderdKVWriter
TEZ-3230. Implement vertex manager and edge manager of cartesian product edge.
TEZ-3395. Refactor ShuffleVertexManager to make parts of it re-usable in other plugins.
TEZ-3382. Tez analyzer: Should be resilient to new counters
TEZ-3379. Tez analyzer: Move sysout to log4j
TEZ-3333. Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
TEZ-3303. Have ShuffleVertexManager consume more precise partition stats.
TEZ-3329. Tez ATS data is incomplete for a vertex which fails or gets killed before initialization
TEZ-3327. ATS Parser: Populate config details available in dag
TEZ-3325. Flaky test in TestDAGImpl.testCounterLimits.
TEZ-3313. ATSFileParser : Wrong args passed in VersionInfo
TEZ-3288. Tez UI: Display more details in the error bar
TEZ-3216. Add support for more precise partition stats in VertexManagerEvent.
TEZ-3295. TestOrderedWordCount should handle relative input/output paths.
TEZ-3292. Tez UI: UTs breaking with timezone change
TEZ-2846. Flaky test: TestCommit.testVertexCommit_OnDAGSuccess.
TEZ-3264. Tez UI: UI discrepancies
TEZ-3289. Tez Example MRRSleep job does not set Staging dir correctly on secure cluster.
TEZ-3276. Tez Example MRRSleep job fails when tez.staging-dir fs is not same as default FS.
TEZ-3063. Tez UI: Display Input, Output, Processor, Source and Sink configurations under a vertex
TEZ-3206. Have unordered partitioned KV output send partition stats via VertexManagerEvent.
TEZ-3255. Tez UI: Hide swimlane while displaying running DAGs from old versions of Tez
TEZ-3254. Tez UI: Consider downloading Hive/Pig explain plans
TEZ-3086. Tez UI: Backward compatibility changes
TEZ-3245. Data race between addKnowInput and clearAndGetOnepartition of InputHost.
TEZ-3193. Deadlock in AM during task commit request.
TEZ-3233. Tez UI: Have LLAP information reflect in Tez UI
TEZ-3203. DAG hangs when one of the upstream vertices has zero tasks
TEZ-3207. Add support for fetching multiple partitions from the same source task to UnorderedKVInput.
TEZ-3232. Disable randomFailingInputs in testFaulttolerance to unblock other tests.
TEZ-3227. Tez UI: Replace UI1 with UI2
TEZ-3228. Update version in master to 0.9.0.
TEZ-3334. Tez Custom Shuffle Handler:
TEZ-3713. Allow dag level deletion in cases where containers are reused
TEZ-3712. Use Local FileContext for deleting dag level directories
TEZ-3633. Implement keep-alive timeout in tez shuffle handler
TEZ-3740. Clean up TEZ-3334-CHANGES.txt
TEZ-3735. Test failures in TestTaskAttempt and TestAMContainerMap
TEZ-3726. Clean up DeletionTracker's reflection instantiation and provide ContainerLauncher with dagComplete() functionality
TEZ-3725. Cleanup http connections and other unnecessary fields in DAG Deletion tracker classes.
TEZ-3705. Modify DeletionTracker and deletion threads to be initialized only if enabled for tez_shuffle
TEZ-3702. Tez shuffle jar includes service loader entry for ClientProtocolProvider but not the corresponding class
TEZ-3685. ShuffleHandler completedInputSet off-by-one error
TEZ-3684. Incorporate first pass non-essential TEZ-3334 pre-merge feedback
TEZ-3682. Pass parameters instead of configuration for changes to support tez shuffle handler
TEZ-3628. Give Tez shuffle handler threads custom names
TEZ-3621. Optimize the Shuffle Handler content length calculation for keep alive
TEZ-3620. UnorderedPartitionedKVOutput is missing the shuffle service config in the confKeys set
TEZ-3618. Shuffle Handler Loading cache equality tests always results is false
TEZ-3612. Tez Shuffle Handler Content length does not match actual
TEZ-3608. Fetcher can hang if copyMapOutput/fetchInputs returns early
TEZ-3606. Fix debug log for empty partitions to the expanded partitionId in the Composite case
TEZ-3604. Remove the compositeInputAttemptIdentifier from remaining list upon fetch completion in the Ordered case
TEZ-3599. Unordered Fetcher can hang if empty partitions are present
TEZ-3596. Number of Empty DME logged for Composite fetch is too high
TEZ-3597. Composite Fetch hangs on certain DME empty events.
TEZ-3595. Composite Fetch account error for disk direct
TEZ-3590. Remove google.protobuf from the tez-auxservices shaded jar
TEZ-3587. Fetcher fetchInputs() can NPE on srcAttempt due to missing entry in pathToAttemptMap
TEZ-3586. Remove fusesource.leveldbjni from the tez-auxservices shaded jar
TEZ-3532. Backport MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log
TEZ-3563. Tez Shuffle Handler logging fails to initialize
TEZ-3564. TezConfiguration#TEZ_DELETION_TRACKER_CLASS has the wrong config key-name
TEZ-3557. TEZ-3362 causes TestContainerLauncherWrapper#testDelegation to fail
TEZ-3361. Fetch Multiple Partitions from the Shuffle Handler
TEZ-3509. Make DAG Deletion path based
TEZ-3480. Port MAPREDUCE-6763 to Tez ShuffleHandler
TEZ-3362. Delete intermediate data at DAG level for Shuffle Handler
TEZ-3360. Tez Custom Shuffle Handler Documentation
TEZ-3411. TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler port
TEZ-3412. Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString()
TEZ-3410. ShuffleHandler should use Path.SEPARATOR instead of '/'
TEZ-3408. Allow Task Output Files to reside in DAG specific directories for Custom Shuffle Handler
TEZ-3238. Shuffle service name should be configureable and should not be hardcoded to ‘mapreduce_shuffle’
TEZ-3390. Package Shuffle Handler as a shaded uber-jar
TEZ-3393. Remove extra jetty dependency from Shuffle Handler
TEZ-3378. Move Shuffle Handler configuration into the Tez namespace
TEZ-3377. Remove ShuffleHandler dependency on mapred.FadvisedChunkedFile and mapred.FadvisedFileRegion
TEZ-3380. Shuffle Handler: Replace primitive wrapper's valueOf method with parse* method to avoid unnecessary boxing/unboxing
TEZ-3355. Tez Custom Shuffle Handler POC
Release 0.8.6: Unreleased
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3007. Use AppFinalState.ENDED when unregistering with the RM in session mode
TEZ-3679. Minor ASF header issues.
TEZ-3678. The command "hadoop dfs" should be replaced by "hadoop fs" in install markdown.
TEZ-3677. by-laws markdown has an incorrect license header.
TEZ-3671. TestCompositeDataMovementEvent has a misplaced Apache license header.
TEZ-3672. Remove duplicate Apache license headers.
TEZ-3561. Fix wrong tez tarball name in install.md.
Release 0.8.5: 2016-03-13
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3709. TezMerger is slow for high number of segments
TEZ-3719. DAGImpl.computeProgress slows down dispatcher and ipc threads
TEZ-3616. TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
TEZ-3644. Cleanup container list stored in AMNode.
TEZ-3643. Long running AMs can go out of memory due to retained AMContainer instances.
TEZ-3624. Split multiple calls on the same line in TaskCommunicatorContextImpl.
TEZ-3550. Provide access to sessionId/dagId via DagClient.
TEZ-3609. Improve ATSv15 performance for DAG entities read calls.
TEZ-3244. Allow overlap of input and output memory when they are not concurrent
TEZ-3601. Add another HistoryLogLevel to suppress TaskAttempts at specific levels
TEZ-3582. Exception swallowed in PipelinedSorter causing incorrect results.
TEZ-3462. Task attempt failure during container shutdown loses useful container diagnostics
TEZ-3574. Container reuse won't pickup extra dag level local resource.
TEZ-3566. Avoid caching fs isntances in TokenCache after a point.
TEZ-3568. Update SecurityUtils configuration to pick user provided configuration.
TEZ-3559. TEZ_LIB_URIS doesn't work with schemes different than the defaultFS
TEZ-3549. TaskAttemptImpl does not initialize TEZ_TASK_PROGRESS_STUCK_INTERVAL_MS correctly
TEZ-3537. ArrayIndexOutOfBoundsException with empty environment variables/Port YARN-3768 to Tez
TEZ-3507. Task logs link when editing url from one task to another.
TEZ-3536. NPE in WebUIService start when host resolution fails.
TEZ-3534. Differentiate thread names on Fetchers, minor changes to shuffle shutdown code.
TEZ-3491. Tez job can hang due to container priority inversion.
TEZ-3533. ShuffleScheduler should shutdown threadpool on exit.
TEZ-3493. DAG submit timeout cannot be set to a month
TEZ-3505. Move license to the file header for TezBytesWritableSerialization
TEZ-3486. COMBINE_OUTPUT_RECORDS/COMBINE_INPUT_RECORDS are not correct
TEZ-3097. Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess.
TEZ-3487. Improvements in travis yml file to get builds to work.
TEZ-3483. Create basic travis yml file for Tez.
TEZ-3437. Improve synchronization and the progress report behavior for Inputs from TEZ-3317.
TEZ-3317. Speculative execution starts too early due to 0 progress.
TEZ-3452. Auto-reduce parallelism calculation can overflow with large inputs
TEZ-3439. Tez joinvalidate fails when first input argument size is bigger than the second.
TEZ-3464. Fix findbugs warnings in tez-dag mainLoop
TEZ-3330. Propagate additional config parameters when running MR jobs via Tez.
TEZ-3335. DAG client thinks app is still running when app status is null
TEZ-3460. Fix precommit release audit warning
TEZ-3368. NPE in DelayedContainerManager
TEZ-3440. Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs
TEZ-3429. Set reconfigureDoneTime on VertexConfigurationDoneEvent properly.
TEZ-3000. Fix TestContainerReuse.
TEZ-3436. Check input and output count before start in MapProcessor.
TEZ-3426. Second AM attempt launched for session mode and recovery disabled for certain cases
TEZ-3326. Display JVM system properties in AM and task logs.
TEZ-3009. Errors that occur during container task acquisition are not logged.
TEZ-2852. TestVertexImpl fails due to race in AsyncDispatcher.
TEZ-3413. ConcurrentModificationException in HistoryEventTimelineConversion for AppLaunchedEvent.
TEZ-3352. MRInputHelpers getStringProperty() should not fail if property value is null.
TEZ-3409. Log dagId along with other information when submitting a dag.
TEZ-3384. Fix TestATSV15HistoryLoggingService::testDAGGroupingGroupingEnabled unit test.
TEZ-3376. Fix groupId generation to account for dagId starting with 1.
TEZ-3359. Add granular log levels for HistoryLoggingService.
TEZ-3374. Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
TEZ-3358. Support using the same TimelineGroupId for multiple DAGs.
TEZ-3357. Change TimelineCachePlugin to handle DAG grouping.
TEZ-3348. NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat.
TEZ-3356. Fix initializing of stats when custom ShuffleVertexManager is used.
TEZ-3329. Tez ATS data is incomplete for a vertex which fails or gets killed before initialization.
TEZ-3235. Modify Example TestOrderedWordCount job to test the IPC limit for large dag plans.
TEZ-3337. Do not log empty fields of TaskAttemptFinishedEvent to avoid confusion.
TEZ-1248. Reduce slow-start should special case 1 reducer runs.
TEZ-3370. Tez UI: Display the log link as N/A if the app does not provide a log link
TEZ-3398. Tez UI: Bread crumb link to Application from Application details dag/configuration tab is broken
TEZ-3433. Tez UI: Searching using wrong ID causes error in all DAGs page
TEZ-3419. Tez UI: Applications page shows error, for users with only DAG level ACL permission
TEZ-3347. Tez UI: Vertex UI throws an error while getting vertexProgress for a killed Vertex
TEZ-3546. Tez UI: On sorting asc - Not Available must be at the top
Release 0.8.4: 2016-07-08
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3323. Update licese and notice for xml-apis jar. Also update year in notice to 2016.
TEZ-3223. Support a NullHistoryLogger to disable history logging if needed.
TEZ-3286. Allow clients to set processor reserved memory per vertex (instead of per container).
TEZ-3293. Fetch failures can cause a shuffle hang waiting for memory merge that never starts.
TEZ-3314. Double counting input bytes in MultiMRInput.
TEZ-3308. Add counters to capture input split length.
TEZ-3302. Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout.
TEZ-3291. Optimize splits grouping when locality information is not available.
TEZ-3305. TestAnalyzer fails on Hadoop 2.7.
TEZ-3304. TestHistoryParser fails with Hadoop 2.7.
TEZ-3296. Tez job can hang if two vertices at the same root distance have different task requirements
TEZ-3294. DAG.createDag() does not clear local state on repeat calls.
TEZ-3297. Deadlock scenario in AM during ShuffleVertexManager auto reduce.
TEZ-3296. Tez fails to compile against hadoop 2.8 after MAPREDUCE-5870
TEZ-3290. Set full task attempt id string in MRInput configuration object.
TEZ-3278. Hide Swimlane from Tez UI
TEZ-3280. LOG MRInputHelpers split generation message as INFO
TEZ-909. Provide support for application tags
TEZ-2769. TEZ-UI Hive SQL is only displayed to line 11
TEZ-3257. Fix flaky test TestUnorderedPartitionedKVWriter.
TEZ-3240. Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars.
TEZ-3237. Corrupted shuffle transfers to disk are not detected during transfer
TEZ-3246. Improve diagnostics when DAG killed by user
TEZ-3258. Jvm Checker does not ignore DisableExplicitGC when checking JVM GC options.
TEZ-3256. [Backport HADOOP-11032] Remove Guava Stopwatch dependency
TEZ-2342. Reduce bytearray copy with TezEvent Serialization and deserialization
TEZ-3251. Allow ability to add custom counters to TaskRunner2Callable.
TEZ-3250. TezTaskRunner2 should accept ExecutorService.
TEZ-3193. Deadlock in AM during task commit request.
TEZ-3203. DAG hangs when one of the upstream vertices has zero tasks
TEZ-3219. Allow service plugins to define log locations link for remotely run task attempts.
TEZ-3224. User payload is not initialized before creating vertex manager plugin.
TEZ-3226. Tez UI 2: All DAGs UX improvements.
TEZ-3077. TezClient.waitTillReady should support timeout.
TEZ-3202. Reduce the memory need for jobs with high number of segments
TEZ-3165. Allow Inputs/Outputs to be initialized serially, control processor initialization relative to Inputs/Outputs
TEZ-3214. Tez UI 2: Pagination in All DAGs
TEZ-3210. Tez UI 2: license should account for numeral, more-js, loader.js , etc
TEZ-3087. Tez UI 2: Add log links in task & attempt details page
TEZ-3146. Tez UI 2: CSS & JS assets in the UI must be minified
TEZ-3259. Tez UI: Build issue - File saver package is not working well with bower
TEZ-3262. Tez UI : zip.js is not having a bower friendly versioning system
TEZ-3281. Tez UI: Swimlane improvements
TEZ-3318. Tez UI: Polling is not restarted after RM recovery
Release 0.8.3: 2016-04-14
INCOMPATIBLE CHANGES
TEZ-3180. Update master docs to declare hadoop-2.6.x as a minimum requirement.
TEZ-3029. Add an onError method to service plugin contexts.
TEZ-3120. Remove TaskCommContext.getCurrentDagName, Identifier.
TEZ-3183. Change the taskFailed method on plugin contexts to specify the type of failure.
TEZ-3199. Rename getCredentials in TaskCommunicatorContext to be less confusing.
ALL CHANGES:
TEZ-3188. Move tez.submit.hosts out of TezConfiguration to TezConfigurationConstants.
TEZ-3194. Tez UI: Swimlane improve in-progress experience.
TEZ-3196. java.lang.InternalError from decompression codec is fatal to a task during shuffle
TEZ-3161. Allow task to report different kinds of errors - fatal / kill.
TEZ-3177. Non-DAG events should use the session domain or no domain if the data does not need protection.
TEZ-3192. IFile#checkState creating unnecessary objects though auto-boxing
TEZ-3173. Update Tez AM REST APIs for more information for each vertex.
TEZ-3108. Add support for external services to local mode.
TEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.
TEZ-2967. Vertex start time should be that of first task start time in UI
TEZ-3175. Add tez client submit host
TEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed tasks.
TEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state
TEZ-2936. Create ATS implementation that enables support for YARN-4265 (ATS v1.5)
TEZ-3148. Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt.
TEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.
TEZ-3155. Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits.
TEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch
TEZ-3140. Reduce AM memory usage during serialization
TEZ-2756. MergeManager close should not try merging files on close if invoked after a shuffle exception.
TEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the application.
TEZ-3115. Shuffle string handling adds significant memory overhead
TEZ-3151. Expose DAG credentials to plugins.
TEZ-3149. Tez-tools: Add username in DagInfo.
TEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier.
TEZ-3147. Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory
TEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout
TEZ-3128. Avoid stopping containers on the AM shutdown thread.
TEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException
TEZ-3114. Shuffle OOM due to EventMetaData flood
TEZ-1911. MergeManager's unconditionalReserve() should check for memory limits before allocating.
TEZ-3102. Fetch failure of a speculated task causes job hang
TEZ-3124. Running task hangs due to missing event to initialize input in recovery.
TEZ-3135. tez-ext-service-tests, tez-plugins/tez-yarn-timeline-history and tez-tools/tez-javadoc-tools missing dependencies.
TEZ-3134. tez-dag should depend on commons-collections4.
TEZ-3126. Log reason for not reducing parallelism
TEZ-3131. Support a way to override test_root_dir for FaultToleranceTestRunner.
TEZ-3067. Links to tez configs documentation should be bubbled up to top-level release page.
TEZ-3123. Containers can get re-used even with conflicting local resources.
TEZ-3117. Deadlock in Edge and Vertex code
TEZ-3103. Shuffle can hang when memory to memory merging enabled
TEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime).
TEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier
TEZ-3090. MRInput should make dagIdentifier, vertexIdentifier, etc available to the InputFormat jobConf.
TEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin.
TEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large jobs.
TEZ-2307. Possible wrong error message when submitting new dag
TEZ-2974. Tez tools: TFileRecordReader in tez-tools should support reading >2 GB tfiles.
TEZ-3081. Update tez website for trademarks feedback.
TEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutputs
TEZ-3079. Fix tez-tfile parser documentation.
TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.
TEZ-3036. Tez AM can hang on startup with no indication of error
TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED
TEZ-2594. Fix LICENSE for missing entries for full and minimal tarballs.
TEZ-3053. Containers timeout if they do not receive a task within the container timeout interval.
TEZ-2898. tez tools : swimlanes.py is broken.
TEZ-2937. Can Processor.close() be called after closing inputs and outputs?
TEZ-3037. History URL should be set regardless of which history logging service is enabled.
TEZ-3032. DAG start time getting logged using system time instead of recorded time in startTime field.
TEZ-3101. Tez UI: Task attempt log link doesn't have the correct protocol.
TEZ-3143. Tez UI 2: Make the build faster
TEZ-3160. Tez UI 2: Swimlane - Create swimlane page & component (sree)
TEZ-3170. Tez UI 2: Swimlane - Display computed events, event bars & dependencies (sree)
TEZ-3152. Tez UI 2: Build fails when run by multiple users or when node_modules is old (sree)
TEZ-3171. Tez UI 2: Swimlane - Tooltip, zoom & redirection (sree)
TEZ-3172. Tez UI: Swimlane - In progress & Shadow (sree)
TEZ-3201. Tez-UI build broken (sree)
TEZ-2980: Tez UI 2 - Umbrella:
TEZ-2982. Tez UI: Create tez-ui2 directory and get a basic dummy page working in ember 2.2
TEZ-3016. Tez UI 2: Make bower dependency silent
TEZ-2983. Tez UI 2: Get ember initializers functional
TEZ-3018. Tez UI 2: Add config.env
TEZ-3019. Tez UI 2: Replace BaseURL with Host
TEZ-2984. Tez UI 2: Create abstract classes
TEZ-3020. Tez UI 2: Add entity blueprint
TEZ-2985. Tez UI 2: Create loader and entity classes
TEZ-3021. Tez UI 2: Add env service & initializer
TEZ-3023. Tez UI 2: Abstract adapter and route
TEZ-3022. Tez UI 2: Add serializer & adapter for timeline server
TEZ-3026. Tez UI 2: Add adapters for RM & AM
TEZ-3027. Tez UI 2: Add header and footer elements
TEZ-2986. Tez UI 2: Implement All DAGs page
TEZ-3038. Tez UI 2: Create DAG details page
TEZ-3039. Tez UI 2: Create all sub-pages for DAG
TEZ-3040. Tez UI 2: Create Vertex details page & sub tables
TEZ-3041. Tez UI 2: Create Task & Attempt details page with sub tables
TEZ-3045. Tez UI 2: Create application details page with DAGs tab
TEZ-3048. Tez UI 2: Make PhantomJS a local dependency for build tests
TEZ-3042. Tez UI 2: Create Counters pages
TEZ-3043. Tez UI 2: Create configurations page
TEZ-3049. Tez UI 2: Add column selector
TEZ-3050. Tez UI 2: Add counter columns
TEZ-3064. Tez UI 2: Add All DAGs filters
TEZ-3059. Tez UI 2: Make refresh functional
TEZ-3070. Tez UI 2: Jenkins build is failing
TEZ-3060. Tez UI 2: Activate auto-refresh
TEZ-3061. Tez UI 2: Display in-progress vertex table in DAG details
TEZ-3069. Tez UI 2: Make error bar fully functional
TEZ-3062. Tez UI 2: Integrate graphical view
TEZ-3058. Tez UI 2: Add download data functionality
TEZ-3084. Tez UI 2: Display caller type and info
TEZ-3080. Tez UI 2: Ensure UI 2 is in-line with UI 1
TEZ-3092. Tez UI 2: Tuneups & Improvements
TEZ-3095. Tez UI 2: Tuneups & Improvements
TEZ-3088. Tez UI 2: Licenses of all the packages used by Tez Ui must be documented
TEZ-2916. Tez UI 2: Show counts of running tasks on the DAG visualization page
TEZ-3125. Tez UI 2: All auto-refresh pages refresh multiple times shortly after application complete
TEZ-3127. Tez UI 2: Release audit is failing
Release 0.8.2: 2016-01-19
INCOMPATIBLE CHANGES
TEZ-3024. Move TaskCommunicator to correct package.
TEZ-2679. Admin forms of launch env settings
TEZ-2948. Stop using dagName in the dagComplete notification to TaskCommunicators.
TEZ-2949. Allow duplicate dag names within session for Tez.
TEZ-604. Revert temporary changes made in TEZ-603 to kill the provided tez session, if running a MapReduce job.
TEZ-2972. Avoid task rescheduling when a node turns unhealthy
ALL CHANGES:
TEZ-2669. Propagation of errors from plugins to the AM for error reporting.
TEZ-2978. Add an option to allow the SplitGrouper to generate node local only groups.
TEZ-2129. Task and Attempt views should contain links to the logs
TEZ-3025. InputInitializer creation should use the dag ugi.
TEZ-3017. HistoryACLManager does not have a close method for cleanup
TEZ-2914. Ability to limit vertex concurrency
TEZ-3011. Link Vertex Name in Dag Tasks/Task Attempts to Vertex
TEZ-3006. Remove unused import in TestHistoryParser.
TEZ-2910. Set caller context for tracing ( integrate with HDFS-9184 ).
TEZ-2976. Recovery fails when InputDescriptor is changed during input initialization.
TEZ-2997. Tez UI: Support searches by CallerContext ID for DAGs
TEZ-2996. TestAnalyzer fails in trunk after recovery redesign
TEZ-2987. TestVertexImpl.testTez2684 fails
TEZ-2995. Timeline primary filter should only be on callerId and not type.
TEZ-2994. LocalProgress in tez-runtime-library missing Apache header, rat check warnings from the new licenses after TEZ-2592 merge.
TEZ-2977. Make HadoopShim selection be overridable for distro-specific implementations.
TEZ-2472. Change slf4j version to 1.7.10.
TEZ-2920. org.apache.tez.client.TestTezClient.testStopRetriesUntilTimeout is flaky.
TEZ-2824. Add javadocs for Vertex.setConf and DAG.setConf.
TEZ-2911. Null location Strings can cause problems with GroupedSplit serialization.
TEZ-2990. Change test-patch.sh to run through all tests, despite failures in upstream modules
TEZ-2798. NPE when executing TestMemoryWithEvents::testMemoryScatterGather.
TEZ-2963. RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication.
TEZ-2966. Tez does not honor mapreduce.task.timeout
TEZ-2979. FlakyTest: org.apache.tez.history.TestHistoryParser.
TEZ-1491. Tez reducer-side merge's counter update is slow.
TEZ-2943. Change shuffle vertex manager to use per vertex data for auto
reduce and slow start
TEZ-2346. TEZ-UI: Lazy load other info / counter data
TEZ-2975. Bump up apache commons dependency.
TEZ-2970. Re-localization in TezChild does not use correct UGI.
TEZ-2968. Counter limits exception causes AM to crash.
TEZ-2960. Tez UI: Move hardcoded url namespace to the configuration file
TEZ-2581. Umbrella for Tez Recovery Redesign
TEZ-2956. Handle auto-reduce parallelism when the
totalNumBipartiteSourceTasks is 0
TEZ-2947. Tez UI: Timeline, RM & AM requests gets into a consecutive loop in counters page without any delay
TEZ-2946. Tez UI: At times RM return a huge error message making the yellow error bar to fill the whole screen
TEZ-2949. Allow duplicate dag names within session for Tez.
TEZ-2952. NPE in TestOnFileUnorderedKVOutput
TEZ-2480. Exception when closing output is ignored.
TEZ-2944. NPE in TestProcessorContext.
TEZ-2948. Stop using dagName in the dagComplete notification to TaskCommunicators.
TEZ-2945. TEZ-2740 addendum to update API with currently supported parameters
TEZ-2933. Tez UI: Load application details from RM when available
TEZ-2908. Tez UI: Errors are logged, but not displayed in the UI when AM fetch fails
TEZ-2923. Tez Live UI counters view empty for vertices, tasks, attempts
TEZ-2924. Framework for Hadoop shims.
TEZ-2935. Add MR slow start translation for ShuffleVertexManager
TEZ-2918. Make progress notifications in IOs
TEZ-2940. Invalid shuffle max slow start setting causes vertex to hang indefinitely
TEZ-2930. Tez UI: Parent controller is not polling at times
TEZ-1670. Add tests for all converter functions in HistoryEventTimelineConversion.
TEZ-2879. While grouping splits, allow an alternate list of preferred locations to be provided per split.
TEZ-2929. Tez UI: Dag details page displays vertices to be running even when dag have completed
TEZ-1976. Findbug warning: Unread field:
org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat$SplitHolder.split
TEZ-2927. Tez UI: Graciously fail when system-metrics-publisher is disabled
TEZ-2915. Tez UI: Getting back to the DAG details page is difficult
TEZ-2895. Tez UI: Add option to enable and disable in-progress
TEZ-2894. Tez UI: Disable sorting for few columns while in progress. Display an alert on trying to sort them
TEZ-2893. Tez UI: Retain vertex info displayed in DAG details page even after completion
TEZ-2878. Tez UI: AM error handling - Make the UI handle cases in which AM returns unexpected/no data
TEZ-2922. Tez Live UI gives access denied for admins
TEZ-2849. Implement Specific Workaround for JDK-8026049 & JDK-8073093.
TEZ-2828. Fix typo in "Shuffle assigned " log statement in shuffle.orderedgrouped.Shuffle.
TEZ-2909. Tez UI: Application link in All DAGs table is disable when applicationhistory is unavailable
TEZ-808. Handle task attempts that are not making progress
TEZ-2553. Tez UI: Tez UI Nits
TEZ-2814. ATSImportTool has a return statement in a finally block
TEZ-2906. Compilation fails with hadoop 2.2.0
TEZ-2900. Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error
TEZ-2244. PipelinedSorter: Progressive allocation for sort-buffers
TEZ-2904. Pig can't specify task specific command opts
TEZ-2888. Make critical path calculation resilient to AM crash
TEZ-2899. Tez UI: DAG getting created with huge horizontal gap in between vertices
TEZ-2907. NPE in IFile.Reader.getLength during final merge operation
TEZ-2903. Stop using proprietary APIs in RPCLoadGen.
TEZ-2882. Consider improving fetch failure handling
TEZ-2850. Tez MergeManager OOM for small Map Outputs
TEZ-1888. Fix javac warnings all over codebase.
TEZ-2886. Ability to merge AM credentials with DAG credentials.
TEZ-2896. Fix thread names used during Input/Output initialization.
TEZ-2866. Tez UI: Newly added columns wont be displayed by default in tables
TEZ-2887. Tez build failure due to missing dependency in pom files.
TEZ-1692. Reduce code duplication between TezMapredSplitsGrouper and TezMapreduceSplitsGrouper.
TEZ-2972. Avoid task rescheduling when a node turns unhealthy
Release 0.8.1-alpha: 2015-10-12
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-2885. Remove counter logs from AMWebController
TEZ-2096. TEZ-UI : Add link to view AM log of finished & running apps
TEZ-2874. Improved logging for caller context.
TEZ-2875. Enable missing tests in TestAnalyzer
TEZ-2781. Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails
TEZ-1788. Allow vertex level disabling of speculation
TEZ-2868. Fix setting Caller Context in Tez Examples.
TEZ-2860. NPE in DAGClientImpl.
TEZ-2855. Fix a potential NPE while routing VertexManager events.
TEZ-2758. Remove append API in RecoveryService after TEZ-1909.
TEZ-2851. Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2859. TestMergeManager.testLocalDiskMergeMultipleTasks failing
TEZ-2858. Stop using System.currentTimeMillis in TestInputReadyTracker.
TEZ-2857. Fix flakey tests in TestDAGImpl.
TEZ-2836. Avoid setting framework/system counters for tasks running in threads.
TEZ-2398. Flaky test: TestFaultTolerance
TEZ-2833. Dont create extra directory during ATS file download
TEZ-2834. Make Tez preemption resilient to incorrect free resource reported
by YARN
TEZ-2775. Improve and consolidate logging in Runtime components.
TEZ-2097. TEZ-UI Add dag logs backend support
TEZ-2812. Preemption sometimes does not respect heartbeats between preemptions
TEZ-814. Improve heuristic for determining a task has failed outputs
TEZ-2832. Support tests for both SimpleHistory logging and ATS logging
TEZ-2827. Increase timeout for TestFetcher testInputAttemptIdentifierMap
TEZ-2774. Improvements and cleanup of logging for the AM and parts of the runtme.
TEZ-2825. Report progress in terms of completed tasks to reduce load on AM for Tez UI
TEZ-2812. Tez UI: Update task & attempt tables while in progress.
TEZ-2786. Tez UI: Update vertex, task & attempt details page while in progress.
TEZ-2612. Support for showing allocation delays due to internal preemption
TEZ-2808. Race condition between preemption and container assignment
TEZ-2807. Log data in the finish event instead of the start event
TEZ-2799. SimpleHistoryParser NPE
TEZ-2643. Minimize number of empty spills in Pipelined Sorter
TEZ-2783. Refactor analyzers to extend TezAnalyzerBase
TEZ-2784. optimize TaskImpl.isFinished()
TEZ-2788. Allow TezAnalyzerBase to parse SimpleHistory logs
TEZ-2782. VertexInfo.getAvgExecutionTimeInterval throws NPE when task does not have any valid attempts info
TEZ-2778. Improvements to handle multiple read errors with complex DAGs
TEZ-2768. Log a useful error message when the summary stream cannot be closed when shutting
down an AM.
TEZ-2745. ClassNotFoundException of user code should fail dag
TEZ-2754. Tez UI: StartTime & EndTime is not displayed with right format in Graphical View
TEZ-2752. logUnsuccessful completion in Attempt should write original finish
time to ATS
TEZ-2755. Fix findbugs warning in TezClient
TEZ-2767. Make TezMxBeanResourceCalculator the default resource calculator.
TEZ-2765. Change Xmlwriter to use defaultValue instead of value tag.
TEZ-2750. Shuffle may not shutdown in case of a fetch failure, causing it to hang.
TEZ-2294. Add tez-site-template.xml with description of config properties.
TEZ-2757. Fix download links for Tez releases.
TEZ-2742. VertexImpl.finished() terminationCause hides member var of the
same name
TEZ-2747. Update master to reflect 0.8.0-alpha release.
TEZ-2662. Provide a way to check whether AM or task opts are valid and error if not.
TEZ-2739. Improve handling of read errors in critical path analyzer
Release 0.8.0-alpha: 2015-09-01
INCOMPATIBLE CHANGES
TEZ-2048. Remove VertexManagerPluginContext.getTaskContainer()
TEZ-2565. Consider scanning unfinished tasks in VertexImpl::constructStatistics to reduce merge overhead.
TEZ-2468. Change the minimum Java version to Java 7.
ALL CHANGES:
TEZ-2749. TaskInfo in history parser should not depend on the apache directory project. Fix master build against hadoop-2.4
TEZ-2748. Fix master build against hadoop-2.2.
TEZ-2743. Fix TezContainerLauncher logging tokens.
TEZ-2708. Rename classes and variables post TEZ-2003 changes.
TEZ-2740. Create a reconfigureVertex alias for deprecated
setVertexParallelism API
TEZ-2690. Add critical path analyser
TEZ-2734. Add a test to verify the filename generated by OnDiskMerge.
TEZ-2732. DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers
TEZ-2687. ATS History shutdown happens before the min-held containers are released
TEZ-2629. LimitExceededException in Tez client when DAG has exceeds the default max counters
TEZ-2730. tez-api missing dependency on org.codehaus.jettison for json.
TEZ-2719. Consider reducing logs in unordered fetcher with shared-fetch option
TEZ-2646. Add scheduling casual dependency for attempts
TEZ-2647. Add input causality dependency for attempts
TEZ-2633. Allow VertexManagerPlugins to receive and report based on attempts
instead of tasks
TEZ-2650. Timing details on Vertex state changes
TEZ-2699. Internalize strings in ATF parser
TEZ-2701. Add time at which container was allocated to attempt
TEZ-2683. TestHttpConnection::testAsyncHttpConnectionInterrupt fails in certain environments.
TEZ-2692. bugfixes & enhancements related to job parser and analyzer.
TEZ-2663. SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.
TEZ-2630. TezChild receives IP address instead of FQDN.
TEZ-2684. ShuffleVertexManager.parsePartitionStats throws IllegalStateException: Stats should be initialized.
TEZ-2172. FetcherOrderedGrouped using List to store InputAttemptIdentifier can lead to some inefficiency during remove() operation.
TEZ-2613. Fetcher(unordered) using List to store InputAttemptIdentifier can lead to some inefficiency during remove() operation.
TEZ-2645. Provide standard analyzers for job analysis.
TEZ-2627. Support for Tez Job Priorities.
TEZ-2623. Fix module dependencies related to hadoop-auth.
TEZ-2464. Move older releases to dist archive.
TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN.
TEZ-2602. Throwing EOFException when launching MR job.
TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source.
TEZ-2616. Fix build warning by undefined version of maven-findbugs-plugin.
TEZ-2588. Improper argument name
TEZ-2575. Handle KeyValue pairs size which do not fit in a single block.
TEZ-2599. Dont send obsoleted data movement events to tasks
TEZ-2542. TezDAGID fromString array length check.
TEZ-2565. Consider scanning unfinished tasks in VertexImpl::constructStatistics to reduce merge overhead.
TEZ-2296. Add option to print counters for tez-examples.
TEZ-2570. Fix license header issue for eps image files.
TEZ-2378. In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size.
TEZ-2558. Upload additional Tez images.
TEZ-2486. Update tez website to include links based on
http://www.apache.org/foundation/marks/pmcs.html#navigation.
TEZ-2548. TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2473. Consider using RawLocalFileSystem in MapOutput.createDiskMapOutput.
TEZ-2538. ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions.
TEZ-2489. Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.
TEZ-2376. Remove TaskAttemptEventType.TA_DIAGNOSTICS_UPDATE
TEZ-2509. YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2506. TestAysncHttpConnection failing.
TEZ-2503. findbugs version isn't reported properly in test-patch report.
TEZ-2198. Fix sorter spill counts.
TEZ-1883. Change findbugs version to 3.x.
TEZ-2440. Sorter should check for indexCacheList.size() in flush().
TEZ-2490. TEZ-2450 breaks Hadoop 2.2 and 2.4 compatability.
TEZ-2450. support async http clients in ordered & unordered inputs.
TEZ-2454. Change FetcherOrderedGroup to work as Callables instead of blocking threads.
TEZ-2466. tez-history-parser breaks hadoop 2.2 compatability.
TEZ-2463. Update site for 0.7.0 release
TEZ-2461. tez-history-parser compile fails with hadoop-2.4.
TEZ-2076. Tez framework to extract/analyze data stored in ATS for specific dag.
TEZ-2436. Tez UI: Add cancel button in column selector.
TEZ-2351. Remove GroupByOrderbyMRRTest example from tez-tests.
TEZ-2419. Inputs/Outputs should inform the Processor about Interrupts when interrupted during a blocking Op.
TEZ-1752. Inputs / Outputs in the Runtime library should be interruptable.
TEZ-1970. Fix javadoc warnings in SortMergeJoinExample.
TEZ-2003: Support for External services CHANGES
TEZ-2019. Temporarily allow the scheduler and launcher to be specified via configuration.
TEZ-2006. Task communication plane needs to be pluggable.
TEZ-2090. Add tests for jobs running in external services.
TEZ-2117. Add a manager for ContainerLaunchers running in the AM.
TEZ-2122. Setup pluggable components at AM/Vertex level.
TEZ-2123. Fix component managers to use pluggable components. (Enable hybrid mode)
TEZ-2125. Create a task communicator for local mode. Allow tasks to run in the AM.
TEZ-2131. Add additional tests for tasks running in the AM.
TEZ-2138. Fix minor bugs in adding default scheduler, getting launchers.
TEZ-2139. Update tez version to 0.7.0-TEZ-2003-SNAPSHOT.
TEZ-2175. Task priority should be available to the TaskCommunicator plugin.
TEZ-2187. Allow TaskCommunicators to report failed / killed attempts.
TEZ-2241. Miscellaneous fixes after last reabse.
TEZ-2283. Fixes after rebase 04/07.
TEZ-2284. Separate TaskReporter into an interface.
TEZ-2285. Allow TaskCommunicators to indicate task/container liveness.
TEZ-2302. Allow TaskCommunicators to subscribe for Vertex updates.
TEZ-2347. Expose additional information in TaskCommunicatorContext.
TEZ-2361. Propagate dag completion to TaskCommunicator.
TEZ-2381. Fixes after rebase 04/28.
TEZ-2388. Send dag identifier as part of the fetcher request string.
TEZ-2414. LogicalIOProcessorRuntimeTask, RuntimeTask, TezTaskRunner should handle interrupts & carry out necessary cleanups.
TEZ-2420. TaskRunner returning before executing the task.
TEZ-2433. Fixes after rebase 05/08
TEZ-2438. tez-tools version in the branch is incorrect.
TEZ-2434. Allow tasks to be killed in the Runtime.
TEZ-2443. TaskRunner2 should call abort, NPEs while cleaning up tasks.
TEZ-2465. Retrun the status of a kill request in TaskRunner2.
TEZ-2471. NPE in LogicalIOProcessorRuntimeTask while printing thread info.
TEZ-2495. Inform TaskCommunicaor about Task and Container termination reasons.
TEZ-2502. Fix for TezTaskRunner2 not killing tasks properly in all situations.
TEZ-2508. rebase 06/01
TEZ-2526. Fix version for tez-history-parser.
TEZ-2621. rebase 07/14
TEZ-2124. Change Node tracking to work per external container source.
TEZ-2004. Define basic interface for pluggable ContainerLaunchers.
TEZ-2005. Define basic interface for pluggable TaskScheduler.
TEZ-2651. Pluggable services should not extend AbstractService.
TEZ-2652. Cleanup the way services are specified for an AM and vertices.
TEZ-2653. Change service contexts to expose a user specified payload instead of the AM configuration.
TEZ-2441. Add tests for TezTaskRunner2.
TEZ-2657. Add tests for client side changes - specifying plugins, etc.
TEZ-2626. Fix log lines with DEBUG in messages, consolidate TEZ-2003 TODOs.
TEZ-2126. Add unit tests for verifying multiple schedulers, launchers, communicators.
TEZ-2698. rebase 08/05
TEZ-2675. Add javadocs for new pluggable components, fix problems reported by jenkins
TEZ-2678. Fix comments from reviews - part 1.
TEZ-2707. Fix comments from reviews - part 2.
TEZ-2713. Add tests for node handling when there's multiple schedulers.
TEZ-2721. rebase 08/14
TEZ-2714. Fix comments from review - part 3.
TEZ-2727. Fix findbugs warnings
TEZ-2670. Remove TaskAttempt holder used within TezTaskCommunicator.
TEZ-2735. rebase 08/21
TEZ-2736. Pre-merge: Update CHANGES.txt and version in branch.
Release 0.7.2: Unreleased
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3696. Jobs can hang when both concurrency and speculation are enabled.
TEZ-3660. Remove CHANGES.txt.
TEZ-3582. Exception swallowed in PipelinedSorter causing incorrect results
TEZ-3559. TEZ_LIB_URIS doesn't work with schemes different than the defaultFS
TEZ-3549. TaskAttemptImpl does not initialize TEZ_TASK_PROGRESS_STUCK_INTERVAL_MS correctly
TEZ-3537. ArrayIndexOutOfBoundsException with empty environment variables/Port YARN-3768 to Tez
TEZ-3507. Task logs link when editing url from one task to another.
TEZ-3536. NPE in WebUIService start when host resolution fails.
TEZ-3493. DAG submit timeout cannot be set to a month
TEZ-3505. Move license to the file header for TezBytesWritableSerialization
TEZ-3486. COMBINE_OUTPUT_RECORDS/COMBINE_INPUT_RECORDS are not correct
TEZ-3437. Improve synchronization and the progress report behavior for Inputs from TEZ-3317.
TEZ-3317. Speculative execution starts too early due to 0 progress.
TEZ-3452. Auto-reduce parallelism calculation can overflow with large inputs
TEZ-3439. Tez joinvalidate fails when first input argument size is bigger than the second.
TEZ-3464. Fix findbugs warnings in tez-dag mainLoop
TEZ-3335. DAG client thinks app is still running when app status is null
TEZ-3460. Fix precommit release audit warning
TEZ-3368. NPE in DelayedContainerManager
TEZ-3440. Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs
TEZ-3426. Second AM attempt launched for session mode and recovery disabled for certain cases
TEZ-3009. Errors that occur during container task acquisition are not logged.
TEZ-3413. ConcurrentModificationException in HistoryEventTimelineConversion for AppLaunchedEvent.
TEZ-3286. Allow clients to set processor reserved memory per vertex (instead of per container).
TEZ-3223. Support a NullHistoryLogger to disable history logging if needed.
TEZ-3293. Fetch failures can cause a shuffle hang waiting for memory merge that never starts.
TEZ-3305. TestAnalyzer fails on Hadoop 2.7.
TEZ-3304. TestHistoryParser fails with Hadoop 2.7.
TEZ-3296. Tez job can hang if two vertices at the same root distance have different task requirements
TEZ-3297. Deadlock scenario in AM during ShuffleVertexManager auto reduce.
TEZ-3296. Tez fails to compile against hadoop 2.8 after MAPREDUCE-5870
TEZ-3278. Hide Swimlane from Tez UI
TEZ-2769. TEZ-UI Hive SQL is only displayed to line 11
TEZ-3280. LOG MRInputHelpers split generation message as INFO
TEZ-3257. Fix flaky test TestUnorderedPartitionedKVWriter.
TEZ-3237. Corrupted shuffle transfers to disk are not detected during transfer
TEZ-3258. Jvm Checker does not ignore DisableExplicitGC when checking JVM GC options.
TEZ-3256. [Backport HADOOP-11032] Remove Guava Stopwatch dependency
TEZ-2342. Reduce bytearray copy with TezEvent Serialization and deserialization
Release 0.7.1: 2016-05-10