Let me start simple; since you have square matrices for both input and filter let me get one dimension. Then you can apply the same for other dimension(s). Imagine your are building fences between trees, if there are N trees, you have to build N-1 fences. Now apply that analogy to convolution layers.
Your output size will be: input size - filter size + 1
Because your filter can only have n-1 steps as fences I mentioned.
Let's calculate your output with that idea.
128 - 5 + 1 = 124
Same for other dimension too. So now you have a 124 x 124 image.
That is for one filter.
If you apply this 40 times you will have another dimension: 124 x 124 x 40