The mtime (modification time) on the directory itself changes when a file or a subdirectory is added, removed or renamed.
Modifying the contents of a file within the directory does not change the directory itself, nor does updating the modified times of a file or a subdirectory. Additionally, adding, removing or renaming files/directories in subdirectories does not propagate up to the directory. If you change the permissions on the directory, the ctime changes but the mtime does not.
A directory is like a file that contains a list of the filenames in it (indeed, you may have accidentally opened a directory in an editor, where you can see it is a mostly-readable list of its contents) -- so it is modified when a file in that directory is added, removed, or renamed.