Scott died in his sleep at home in Las Vegas on Thursday as a result of cardiac arrest following a period of illness, his wife, Jeannie Scott, said in a telephone interview.
Scott became known as a "singer's singer" who was revered by music figures such as Reed, Quincy Jones and Madonna for rendering standards from the American songbook in his distinctive high register.
His unusual voice was a result of being born with a rare genetic condition called Kallmann syndrome, which meant he never passed through puberty.
His early career had its moments of triumph, but seemed to stop more often than it started. He performed as Little Jimmy Scott with the Lionel Hampton Band in the late 1940s and early 1950s, although his name went uncredited on recordings.
Charles signed Scott to his record label in the early 1960s, and accompanied Scott's vocals on the piano for the album "Falling in Love Is Wonderful." The record was quickly pulled from stores in a contract dispute with another label, but would go on to acquire cult acclaim among jazz aficionados.
Attempts over the next few years to revive his recording career were similarly stymied. Scott returned to his birthplace, Cleveland, and took a series of odd jobs, working as a nursing home aide and as a shipping clerk, according to a New York Times Magazine profile from 2000.
Scott's music career was revived in 1991, when he sang at the funeral of his old friend Doc Pomus, a blues songwriter.
Seymour Stein, a record label executive, was at the funeral and signed Scott, who returned to the studio, his voice a little smokier than in his youth, to record a quartet of albums in the 1990s for Sire and Warner Bros Records. Scott's 1992 album, "All the Way," was nominated for a Grammy, and he began touring internationally in his 60s.
Scott said it was not until his 30s that he learned to embrace his pristine voice. "Well, I learned that it was a gift that I was able to sing this way," he told the New York Times Magazine.