You can attach the Debugger to a running process with an attach entry in launch.json, but as far as I can tell, there is no corresponding way to detach the debugger (similar to Visual Studio's Debug > Detach All).
There are a number of scenarios where this can be useful, e.g:
When debugging client-server code you may want to switch the debugger between the client and the server process, without killing any of the processes.
For hot-reloading of DLLs, you may want to detach the debugger so that it releases its file locks, allowing you to rebuild DLLs and PDBs.
For attach, the only option should be detach.

To help investigate why detach is not working for you could you answer the following questions:
It would be nice with a Detach option even if you didn't attach to a running process, but started it using Debug.
VSCode will need to add a way to detach the process in the UI. At the moment, they only support terminating a process.
@WardenGnaw Could you please tell me how can I change the behavior of the stop button so that it can detach the gdb from the process rather than terminate it. I can hit the pause button then input “-exec detach” to do it, I guess there must be somewhere in the code that I can customize the behavior. Can you help me with that? Thanks.
VS Code now has a way to contribute to the debug toolbar.
This feature will need to register a command that sends a disconnect request to the debug adapter.
Most helpful comment
It would be nice with a Detach option even if you didn't attach to a running process, but started it using Debug.