Everything that is not C++ standard, i.e. all Windows headers, all C headers, and anything "old school" take .h. You should use the extension for your own headers too.
Thus, you will want: <windows.h>, <stdio.h>, <stddef.h>, <emmintrin.h> etc...
However, the C++ standard library headers have no extension, so if you use these, you want
<iostream>, <vector>, <map>, <algorithm>, etc...