I've seen videos of people having a clickable window that shows the class and information in Godot. How do I go about getting that?

Press "Search Help" on the top rated that of the script editor (visible in this screenshot).
This could likely be documented in the Editor Manual section.
I think a good place to put this would be in Introduction to Godot's Editor, along with screenshots. I would be happy to make these changes.
I felt silly when I realized I just didn't click the right button, but after that I was able to take off and find everything I needed
I have made the edits on this page and created a pull request. https://github.com/godotengine/godot-docs/pull/3298
The PR was not properly linked, but #3298 did indeed add the information about "Search Help". So, this is resolved now?
Yes, fixed by #3298.
We removed the "Request Docs" button in master / 3.2.2 so the first screenshot might need to be redone, though it's low priority IMO (it's still fully understandable as is).
Most helpful comment
Press "Search Help" on the top rated that of the script editor (visible in this screenshot).
This could likely be documented in the Editor Manual section.