It was however originally only open to men and with women only able to compete in Olympic competition from 1912. It is one of the most popular spectator events at the Games and has the second-highest number of Olympic events (after athletics).

In the first four Olympics, competitions were not held in pools, but rather in open water (1896, the Mediterranean Sea; 1900, the Seine; 1904, an artificial lake; 1906, the Mediterranean). The 1904 Olympics' races were the only ones ever measured in yards, instead of the usual metres. A 100-metre pool was built for the 1908 Olympics and was located in the centre of the main stadium's track and field oval. The 1912 Olympics, held in the Stockholm Harbour, marked the beginning of electrical timing.

The 1924 Olympics were the first to use the standard 50 metre pool with marked lanes. In the freestyle, swimmers originally dove from the pool walls, but diving blocks were incorporated starting at the 1936 Summer Olympics. The flip-turn was developed by the 1950s. Swimming goggles were first allowed in 1976.

Women were first allowed the longer distance of 800 metres freestyle in 1968, which remains the longest distance for women in the swimming pool. Up until then, women were considered to be too "delicate" to swim a race this long, even though the men had had the 1500 metre freestyle from the very early years. Women's teams were first granted the 800 metre freestyle relay race in 1996 even though there had been a men's relay race at that distance since 1912. At six Olympiads, the men had two freestyle relay races, but the women only had one. Now, both sexes have two. The medley relay races (4×100 metres) were not held for men or for women until 1960, but they have continued in every Olympiad since then.

Starting in 2008, both men and women compete in 10 km open water marathon swim events.

The 2020 Olympics will see the introduction for the first time of a 4 x 100m Mixed Medley Relay.