The pilgrims were some of the nearly 2 million Muslims taking part in the hajj, a rite required of able-bodied faithful who can afford it at least once in a lifetime.
Saudi authorities responded quickly to the fire in an apartment building in the Gazza neighborhood and it was extinguished in 10 minutes, a Saudi security official said on condition of anonymity.
Gazza is about 2 kilometers from the Grand Mosque, which houses the sacred black cubic structure known as the Kaaba that Muslims around the world face in prayer five times a day.
The blaze did not interrupt the thousands of pilgrims who converged on the mosque and other nearby holy sites. Turnout was so high that worshippers were forced to stand and pray instead of the traditional bowing on a carpet.
Pilgrims saw and pointed at the black smoke belching from the building, but few were alarmed. A government helicopter was seen hovering above the fire.
In 1997, a fire in the tent city of Mina, near Mecca, killed 340 pilgrims.
Last week Saudi authorities announced that 50,000 security forces had been mobilised to handle the wave of arriving pilgrims.