The =~ operator is a regular expression match operator. This operator is inspired by Perl's use of the same operator for regular expression matching.
The [[ ]] is treated specially by bash; consider that an augmented version of [ ] construct:
[ ] is actually a shell built-in command, which, can actually be implemented as an external command. Look at your /usr/bin, there is most likely a program called "[" there! Strictly speaking, [ ] is not part of bash syntax.
[[ ]] is a shell keyword, which means it is part of shell syntax. Inside this construct, some reserved characters change meaning. For example, ( ) means parenthesis like other programming language (not launching a subshell to execute what's inside the parentheses). Another example is that < and > means less than and greater than, not shell redirection. This allow more "natural" appearance of logical expressions, but it can be confusing for novice bash programmers.
The =~operator is the pattern match operator. It did not exist in the original Bourne shell, when test, or internally [ ], was used for conditionals.
The let command, or [[ ]] internally, has more functionality than test, including pattern matching capabilities. This is why you have to use [[ ]], instead of [ ], when using =~.