One use is for enforcing memory alignment on variables and structure members. For example
float vect[4] __attribute__((aligned(16)));
Will ensure that vect will be placed on a 16 byte memory boundary. I do not know if that is a gcc-ism or more generally applicable.
The compiler will typically only aligned vect on a 4 byte boundary. With 16 byte alignment it can be used directly with SIMD load instructions where you'd load it up into a 128 bit registers that allows addition, subtraction, dot products and all manner of vector operations.
Sometimes you want alignment so that a structure can be directly overlaid onto memory-mapped hardware registers. Or it has to be aligned so the hardware can write into it directly used a direct memory access (DMA) mechanism.
Why is it used in C programs? To limit their portability.
It begins with a double-underscore, so it's in the implementor's namespace - it's not something defined by the language standard, and each compiler vendor is free to use it for any purpose whatsoever.
Edit: Why is it used in GNU C programs? See the other answers that address this.
For what GCC and GCC-compatible compilers use __attribute__ most other compilers use #pragma directives.
I think GCC's solution is better since the required behavior of an unrecognised #pragma is to ignore it, whereas if you use a compiler that does not understand an __attribute__ specification, it will not compile - which is generally better, since you then know what you need to port.
Attribute specifications are used to specify aspects of types, data, and functions such as storage and alignment that cannot be specified using C. Often these are target specific, mostly they are non-portable, certainly between compilers, and often between targets. Avoid their use except where it is absolutely necessary to use correct functions of code.