Our Gemara tells us that though the Cohen Godol wore בגדי לבן on Yom Kippur for services related to the Holy of Holies, which were really the same as the Cohen Hedyot. they would not be recycled for Cohen hedyot use during the year.
On the topic of Bigdei Lavan, we find that one time in history a non-cohen wore the white priestly clothes and performed the service in the inner sanctuary. Moshe served as Cohen Godol during the seven inaugural days of the Mishkan. He too wore Bigdei Lavan, yet there were some physical and symbolic differences.
Maharal (מהר״ל גבורת השם כח):
The Gemara (Taanis 11a) tells us that Moshe’s Bigdei Lavan did not have a hem, while the traditional Cohen’s cloak did have a hem. This is an interesting detail, but what does it mean?
The Maharal explains this based on another famous Gemara (Niddah 30b):
A fetus is taught the entire Torah while in the womb, as it is stated: “And He taught me and said to me: Let your heart hold fast My words; keep My commandments, and live” (Proverbs 4:4). And it also states: “As I was in the days of my youth, when the converse of God was upon my tent” (Job 29:4)...And once the fetus emerges into the airspace of the world, an angel comes and slaps it on its mouth, causing it to forget the entire Torah, as it is stated: “Sin crouches at the entrance” (Genesis 4:7), i.e., when a person enters the world he is immediately liable to sin due to his loss of Torah knowledge.
By the way, apparently the Greeks had a similar mystical tradition, perhaps learned from us, or perhaps a common mystical tradition from Adam HaRishon. Meno's slave is a character in the Socratic dialogue Meno (82b–85b), which was written by Plato. And through an interesting mathematical experiment with the so-called ignorant slave, Socrates proved that the soul already knows everything and all instruction is merely a re-learning.
The Maharal explains this loss of knowledge at the time of birth as not cruelty on God’s part, but rather a necessary effect of physical biological life. The essence of human consciousness is speech. In order to be a conscious functional human being, the overwhelming and complete knowledge of all and everything that the soul possesses via attachment to God must be mitigated. Getting hit on the mouth by the angel, is metaphoric for, the mouth, i.e. human speech and consciousness, as the factor that leads to the soul’s forgetting of complete knowledge. This is necessary for human consciousness. Otherwise we would be overwhelmed and bask in God’s glory and knowledge of all. Parayzed by greatness.
Maharal goes on to say a remarkable insight. Moshe had a speech impediment because his soul was on such a lofty level and he remained still attached to it. So much so, that the regular day-to-day temporal functions of speech and engagement with people in a linear fashion eluded him. Interestingly, Ralbag makes a similar observation (Shemos 4:10). Thus, Moshe‘s speech impediment was not a deformity, but rather that his soul and body never fully landed and he never became fully human.
This is what is represented in the cloak without a hem. The hem represents the final stitches, the makeh be-patish of a garment. Symbolically, a cloak represents physicality and the body. Just as a Cloak covers the body, so too the soul is covered by physicality. Therefore, the white cloak worn by Moshe, did not have a hem, to express the truth that his body did not have that final formation, and his soul remained primary.
Translations Courtesy of Sefaria