The defending champions, who lost 2-5 against Japan in the pool stage, scored two eye-catching field goals through Manjeet (8th minute) and Pawan Rajbhar (35th) to register a narrow win.
Japan's lone goal came from a penalty corner through Takuma Niwa in the 18th minute.
Japan threatened India initially and secured a penalty corner as early as in the opening minute but the Indians thwarted the danger.
The Indians grew in confidence and got their footing as the match progressed.
It was a moment of brilliance from Manjeet that handed India the lead. He made a brilliant solo run down the left flank after receiving a pass from Pawan Rajbhar and used 3D skills to perfection to beat Takashi Yoshikawa under the Japanese bar.
Maninder Singh earned India a penalty corner in the 13th minute but Nilam Sanjeep Xess' effort was saved by the Japanese defence.
As was the case in the first quarter, the Japanese started the second 15 minutes on a promising note and secured a penalty corner in the 18th minute from which Takuma Niwas scored from a rebound after Ken Nagayoshi's initial flick was saved by Indian goalkeeper Karkera.
Pepped up by the goal, Japan continued to put pressure on the Indian defence and earned two more penalty corners, which the Indian defence kept at bay.
Minutes later, Karthi Selvam missed a golden chance as he failed to get past Japanese goalie Yoshikawa from one-one-one situation.
Rajbhar restored India's lead five minutes into the second half, slotting home from close range after being set up by Uttam Singh's brilliant stick work.
Five minutes later, India conceded another penalty corner but the Japanese failed to capitalise on the chance.
Trailing by a goal, the Japanese mounted relentless pressure on the Indian citadel but the Birendra Lakra-led backline stood firm to keep their lead intact.
Just three minutes from the final hooter, Japan secured a penalty corner but the Indians defended in numbers to register the win.
India will play Malaysia in their next Super 4 stage match on Sunday.
Meanwhile, in the other Super 4 match of the day, South Korea and Malaysia played out a 2-2 draw.
In the Super 4s, India, Japan, South Korea and Malaysia will play each other once with the top two teams progressing to the final.