EMR master instance is not reachable


I faced a rare issue today.  My EMR cluster was not resizing.
Apparently, the master node was facing issue in communicating with EMR.
The cluster status showed "Master instance is not reachable. Please check your master instance status". But I was able to login into master node and use hadoop and HDFS without any issues. Just that the cluster was not resizing.

Searching on the internet just gave one relevant result.
https://forums.aws.amazon.com/thread.jspa?messageID=695687

Although it mentioned that the issue was with the emr 3.x amis and my cluster was a 4.x, I thought this was the same issue. For verifying I checked on my other clusters if any such service was running on them.

$ ps awux | grep instancecontroller
hadoop    2389  0.4  2.0 3584132 309980 ?      Sl   Apr13  97:41 /etc/alternatives/jre/bin/java -Xmx1024m -XX:OnOutOfMemoryError=kill -9 %p -XX:MinHeapFreeRatio=10 -server -cp /usr/share/aws/emr/instance-controller/lib/*:/home/hadoop/conf -Dlog4j.defaultInitOverride aws157.instancecontroller.Main
hadoop   18522  0.0  0.0 110460  2080 pts/2    S+   19:08   0:00 grep --color=auto instancecontroller

So I gave it a try using the following command as -XX "kill -9 %p" was giving error

$ /etc/alternatives/jre/bin/java -Xmx1024m -XX:MinHeapFreeRatio=10 -server -cp /usr/share/aws/emr/instance-controller/lib/*:/home/hadoop/conf -Dlog4j.defltInitOverride aws157.instancecontroller.Main &

Alas! the status on EMR console went ok again. All good till now.
Although later I figured out the downsizing still doesn't work. I terminated all TASK nodes manually and CORE nodes too (Always keep data in S3 instead of HDFS). The dfs is showing missing blocks but who cares.
I tried adding a TASK node and it worked. 

Comments

Sohil Jain said…
Thank you Vipul. Your forum helped me resolve a production issue today.

There is an error in the command above. Please change defltInitOverride to defaultInitOverride.
May 20, 2019 at 11:08 PM
Unknown said…
@sohil : his command was right i was able to get it correctly
October 30, 2019 at 2:13 AM
Unknown said…
Which command is correct may depend on version. I used:

sudo /etc/init.d/instance-controller start

that file contained the correct command to run for me.
December 1, 2020 at 3:44 AM
Post a Comment

Popular posts from this blog

Adding jar files to weka

Recently, I had to implement a machine learning algorithm called MLRules. I found the code on the author's webpage but there was a big problem. It was in Java using the WEKA environment. I had never worked in the Java before. Above that, I had another alien named 'weka'. Initially everything seemed out of the world. I tried different things, searched on google, read wiki after wiki but there doesn't seem to be a working solution of adding new classifiers in form of .jar files to run in weka. My search on google began with query 'adding jar files to weka'. I searched and searched till the query became 'cannot add jar file to weka'. I was nearly ending an unsuccessful pursuit of the 'WEKA' when one of the results came up from the 'google code' site about adding thier code to weka. The steps were well documented. I tried working it out and ALAS!!! it started working. It took a week's time just to do a simple thing. I ran the code and it ...
Read more

Writing BOLD text in Gmail Chat

Did you ever felt the need to write something in BOLD when chatting on Gmail. Now, here's a simple trick. Just add asterisk on both sides of the *word*. Want to bold a phrase or complete text. Just put * in the beginning and at the end *. Its as simple as that!!!!!! Now be BOLD and CHAT .
Read more