Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Job 2: Failed #4

Open
mflipe opened this issue May 8, 2021 · 0 comments
Open

Job 2: Failed #4

mflipe opened this issue May 8, 2021 · 0 comments

Comments

@mflipe
Copy link

mflipe commented May 8, 2021

Job 2 Failed

  • Compiled with SBT 1.4 on windows
  • No changes to the source code
  • Executed on Ubuntu WSL 2

LOG:

$ hadoop jar hw2_dblp_mapred.jar 2 input
2021-05-07 18:22:02,295 INFO hw2.RunJobs$: Starting up MapReduce job..
2021-05-07 18:22:02,354 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2021-05-07 18:22:02,583 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-05-07 18:22:02,669 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hduser/.staging/job_1620420770486_0002
2021-05-07 18:22:05,246 INFO input.FileInputFormat: Total input files to process : 1
2021-05-07 18:22:05,810 INFO mapreduce.JobSubmitter: number of splits:24
2021-05-07 18:22:06,035 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1620420770486_0002
2021-05-07 18:22:06,035 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-05-07 18:22:06,158 INFO conf.Configuration: resource-types.xml not found
2021-05-07 18:22:06,158 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-05-07 18:22:06,204 INFO impl.YarnClientImpl: Submitted application application_1620420770486_0002
2021-05-07 18:22:06,233 INFO mapreduce.Job: The url to track the job: http://[MYUSER].localdomain:8088/proxy/application_1620420770486_0002/
2021-05-07 18:22:06,234 INFO mapreduce.Job: Running job: job_1620420770486_0002
2021-05-07 18:22:12,284 INFO mapreduce.Job: Job job_1620420770486_0002 running in uber mode : false
2021-05-07 18:22:12,284 INFO mapreduce.Job:  map 0% reduce 0%
2021-05-07 18:22:36,479 INFO mapreduce.Job:  map 1% reduce 0%
2021-05-07 18:22:48,545 INFO mapreduce.Job:  map 2% reduce 0%
2021-05-07 18:23:06,653 INFO mapreduce.Job:  map 3% reduce 0%
2021-05-07 18:23:18,751 INFO mapreduce.Job:  map 4% reduce 0%
2021-05-07 18:23:30,821 INFO mapreduce.Job:  map 5% reduce 0%
2021-05-07 18:23:42,913 INFO mapreduce.Job:  map 6% reduce 0%
2021-05-07 18:24:01,016 INFO mapreduce.Job:  map 7% reduce 0%
2021-05-07 18:24:13,079 INFO mapreduce.Job:  map 8% reduce 0%
2021-05-07 18:24:25,130 INFO mapreduce.Job:  map 9% reduce 0%
2021-05-07 18:24:37,168 INFO mapreduce.Job:  map 10% reduce 0%
2021-05-07 18:24:49,211 INFO mapreduce.Job:  map 11% reduce 0%
2021-05-07 18:25:01,260 INFO mapreduce.Job:  map 12% reduce 0%
2021-05-07 18:25:19,327 INFO mapreduce.Job:  map 13% reduce 0%
2021-05-07 18:25:31,393 INFO mapreduce.Job:  map 14% reduce 0%
2021-05-07 18:25:43,422 INFO mapreduce.Job:  map 15% reduce 0%
2021-05-07 18:25:55,465 INFO mapreduce.Job:  map 16% reduce 0%
2021-05-07 18:25:59,493 INFO mapreduce.Job:  map 19% reduce 0%
2021-05-07 18:26:06,523 INFO mapreduce.Job:  map 20% reduce 0%
2021-05-07 18:26:07,534 INFO mapreduce.Job:  map 22% reduce 0%
2021-05-07 18:26:18,576 INFO mapreduce.Job:  map 24% reduce 0%
2021-05-07 18:26:19,581 INFO mapreduce.Job:  map 25% reduce 0%
2021-05-07 18:26:28,619 INFO mapreduce.Job:  map 26% reduce 0%
2021-05-07 18:26:35,660 INFO mapreduce.Job:  map 26% reduce 8%
2021-05-07 18:26:38,680 INFO mapreduce.Job: Task Id : attempt_1620420770486_0002_m_000010_0, Status : FAILED
Error: java.util.NoSuchElementException: head of empty list
        at scala.collection.immutable.Nil$.head(List.scala:629)
        at scala.collection.immutable.Nil$.head(List.scala:628)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:31)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:8)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2021-05-07 18:26:47,724 INFO mapreduce.Job:  map 27% reduce 8%
2021-05-07 18:26:58,778 INFO mapreduce.Job: Task Id : attempt_1620420770486_0002_m_000010_1, Status : FAILED
Error: java.util.NoSuchElementException: head of empty list
        at scala.collection.immutable.Nil$.head(List.scala:629)
        at scala.collection.immutable.Nil$.head(List.scala:628)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:31)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:8)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2021-05-07 18:27:09,836 INFO mapreduce.Job:  map 28% reduce 8%
2021-05-07 18:27:27,913 INFO mapreduce.Job:  map 29% reduce 8%
2021-05-07 18:27:45,966 INFO mapreduce.Job:  map 30% reduce 8%
2021-05-07 18:28:04,032 INFO mapreduce.Job:  map 31% reduce 8%
2021-05-07 18:28:21,072 INFO mapreduce.Job:  map 32% reduce 8%
2021-05-07 18:28:39,139 INFO mapreduce.Job:  map 33% reduce 8%
2021-05-07 18:28:59,777 INFO mapreduce.Job:  map 34% reduce 8%
2021-05-07 18:29:17,832 INFO mapreduce.Job:  map 35% reduce 8%
2021-05-07 18:29:35,891 INFO mapreduce.Job:  map 36% reduce 8%
2021-05-07 18:29:40,907 INFO mapreduce.Job: Task Id : attempt_1620420770486_0002_m_000009_0, Status : FAILED
Error: java.util.NoSuchElementException: head of empty list
        at scala.collection.immutable.Nil$.head(List.scala:629)
        at scala.collection.immutable.Nil$.head(List.scala:628)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:31)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:8)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2021-05-07 18:29:41,921 INFO mapreduce.Job:  map 33% reduce 8%
2021-05-07 18:29:47,940 INFO mapreduce.Job:  map 34% reduce 8%
2021-05-07 18:29:53,283 INFO mapreduce.Job:  map 37% reduce 8%
2021-05-07 18:29:54,292 INFO mapreduce.Job:  map 37% reduce 10%
2021-05-07 18:30:00,313 INFO mapreduce.Job:  map 37% reduce 11%
2021-05-07 18:30:03,340 INFO mapreduce.Job: Task Id : attempt_1620420770486_0002_m_000010_2, Status : FAILED
Error: java.util.NoSuchElementException: head of empty list
        at scala.collection.immutable.Nil$.head(List.scala:629)
        at scala.collection.immutable.Nil$.head(List.scala:628)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:31)
        at com.samujjwaal.hw2.mappers.VenueOneAuthorMapper.map(VenueOneAuthorMapper.scala:8)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2021-05-07 18:30:06,346 INFO mapreduce.Job:  map 38% reduce 11%
2021-05-07 18:30:12,370 INFO mapreduce.Job:  map 38% reduce 13%
2021-05-07 18:30:19,385 INFO mapreduce.Job:  map 39% reduce 13%
2021-05-07 18:30:25,396 INFO mapreduce.Job:  map 100% reduce 100%
2021-05-07 18:30:27,402 INFO mapreduce.Job: Job job_1620420770486_0002 failed with state FAILED due to: Task failed task_1620420770486_0002_m_000010
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2021-05-07 18:30:27,463 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=38575671
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1207962496
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=27
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
                HDFS: Number of bytes read erasure-coded=0
        Job Counters
                Failed map tasks=5
                Killed map tasks=14
                Killed reduce tasks=1
                Launched map tasks=18
                Launched reduce tasks=1
                Other local map tasks=4
                Data-local map tasks=14
                Total time spent by all maps in occupied slots (ms)=2679662
                Total time spent by all reduces in occupied slots (ms)=245579
                Total time spent by all map tasks (ms)=2679662
                Total time spent by all reduce tasks (ms)=245579
                Total vcore-milliseconds taken by all map tasks=2679662
                Total vcore-milliseconds taken by all reduce tasks=245579
                Total megabyte-milliseconds taken by all map tasks=2743973888
                Total megabyte-milliseconds taken by all reduce tasks=251472896
        Map-Reduce Framework
                Map input records=2364905
                Map output records=417710
                Map output bytes=35339828
                Map output materialized bytes=36191841
                Input split bytes=1017
                Combine input records=0
                Spilled Records=417710
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=25146
                CPU time spent (ms)=2043870
                Physical memory (bytes) snapshot=5030256640
                Virtual memory (bytes) snapshot=23105875968
                Total committed heap usage (bytes)=4161798144
                Peak Map Physical memory (bytes)=567414784
                Peak Map Virtual memory (bytes)=2582102016
        File Input Format Counters
                Bytes Read=1207961479
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
DBLP MapReduce
Status: To do
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mflipe