There's a great list of such instruments here.
The 'instruments' that have been used to create EA music vary widely - here are some obvious and perhaps some not-so-obvious examples (in alphabetical order):
Csound is the primary freeware sound synthesis/processing program for the PC, Mac, Linux, various varieties of Unix and others. Csound is the culmination of a series of programs that have been under continuous development since the early 1960's when Max Mathews wrote the Music 4 program. Barry Vercoe at M.I.T. has been central to the development of the program since 1968. It is now maintained and extended by a loose-knit team of programmers.
Csound has almost unlimited sound generation capabilities, but it is not the easiest program to get to grips with. It also takes a long time to process its score and orchestra input files into the output sound wave files. The recent generations of fast computers are at last making its extensive capabilities available to home users without leaving it to run overnight. Apart from the dedicated few composers prepared to master its complexities, it finds its main use is in the music departments of universities.
The Mark 2 version was delivered to the Columbia-Princeton Electronic Music Center in 1959. Although its synthesis capabilities were somewhat extended, its output was still to disk (RCA was a disk producer, after all!). It was quickly converted to four-track tape output by the engineers at the studio.
The Telharmonium or Dynamophone
In 1897 Thaddeus Cahill patented in the U.S. the first electrically based sound-generation system. This was later known as the Telharmonium or Dynamophone, and was first presented to the public in 1906. As the name Dynamophone suggests, it used modified dynamos running at different speeds to produce alternating electric currents at different audio frequencies. These currents passed via a polyphonic keyboard to a series of telephone receivers fitted with special acoustic horns.
The main problem with this machine was its size and cost - it weighed some 200 tons and cost $200,000! Cahill and the New England Electric Music Company intended to sell telharmoniums to cities and towns throughout America for the transmission of 'Telharmony' to hotels, restaurants, theatres and private homes via the local telephone exchange. Unfortunately the machine seriously interfered with normal telephone calls, and the venture went bust just before the First World War.
Tempophon or "Springer Machine"
These were tape recorders fitted with special rotating heads (rather like a modern video recorder) that could alter the pitch and length of sounds within certain limits. Two of these were in use in the Milan studio in the late 1950s.
Some other relevant links:
List of electronic instruments