Just to clarify, it only vibrates if you have the vibrate function on. So for example, if your phone is on silent and you have 'vibrate on silent' set to On, you'll get that little vibrate pattern. Otherwise it does nothing. The general idea is that if you long press the Home button, you're looking at the screen so don't need an intrusive beep, which was particularly annoying if the phone was on silent.
If you have headphones in or the phone is connected to a Bluetooth device (I think), you will hear the beeps to indicate that Siri has been activated, because you might not be looking at the screen.