| title | description | author | ms.author | ms.reviewer | ms.service | ms.topic | ms.custom | ms.date |
|---|---|---|---|---|---|---|---|---|
Understand and resolve WebHCat errors on HDInsight - Azure |
Learn how to about common errors returned by WebHCat on HDInsight and how to resolve them. |
hrasheed-msft |
hrasheed |
jasonh |
hdinsight |
troubleshooting |
hdinsightactive |
04/14/2020 |
Learn about errors received when using WebHCat with HDInsight, and how to resolve them. WebHCat is used internally by client-side tools such as Azure PowerShell and the Data Lake Tools for Visual Studio.
WebHCat is a REST API for HCatalog, a table, and storage management layer for Apache Hadoop. WebHCat is enabled by default on HDInsight clusters, and is used by various tools to submit jobs, get job status, and so on, without logging in to the cluster.
Several of the errors listed in this document occur because a configured maximum has been exceeded. When the resolution step mentions that you can change a value, use Apache Ambari (web or REST API) to modify the value. For more information, see Manage HDInsight using Apache Ambari
If the following default values are exceeded, it can degrade WebHCat performance or cause errors:
| Setting | What it does | Default value |
|---|---|---|
| yarn.scheduler.capacity.maximum-applications | The maximum number of jobs that can be active concurrently (pending or running) | 10,000 |
| templeton.exec.max-procs | The maximum number of requests that can be served concurrently | 20 |
| mapreduce.jobhistory.max-age-ms | The number of days that job history are retained | 7 days |
HTTP Status code: 429
| Cause | Resolution |
|---|---|
| You've exceeded the maximum concurrent requests served by WebHCat per minute (default 20) | Reduce your workload to ensure that you don't submit more than the maximum number of concurrent requests or increase the concurrent request limit by modifying templeton.exec.max-procs. For more information, see Modifying configuration |
HTTP Status code: 503
| Cause | Resolution |
|---|---|
| This status code usually occurs during failover between the primary and secondary HeadNode for the cluster | Wait two minutes, then retry the operation |
HTTP Status code: 400
| Cause | Resolution |
|---|---|
| Job details have been cleaned up by the job history cleaner | The default retention period for job history is 7 days. The default retention period can be changed by modifying mapreduce.jobhistory.max-age-ms. For more information, see Modifying configuration |
| Job has been killed because of a failover | Retry job submission for up to two minutes |
| An Invalid job ID was used | Check if the job ID is correct |
HTTP Status code: 502
| Cause | Resolution |
|---|---|
| Internal garbage collection is occurring within the WebHCat process | Wait for garbage collection to finish or restart the WebHCat service |
| Time out waiting on a response from the ResourceManager service. This error can occur when the number of active applications goes the configured maximum (default 10,000) | Wait for currently running jobs to complete or increase the concurrent job limit by modifying yarn.scheduler.capacity.maximum-applications. For more information, see the Modifying configuration section. |
Attempting to retrieve all jobs through the GET /jobs call while Fields is set to * |
Don't retrieve all job details. Instead use jobid to retrieve details for jobs only greater than certain job ID. Or, don't use Fields |
| The WebHCat service is down during HeadNode failover | Wait for two minutes and retry the operation |
| There are more than 500 pending jobs submitted through WebHCat | Wait until currently pending jobs have completed before submitting more jobs |
[!INCLUDE troubleshooting next steps]