Have you ever watched a movie or read a book and realized the characters are defiantly not where they say they are? I remember seeing a movie that showed Denver, Colorado as a small ski town in the mountains. Not only did it break my suspension of disbelief in the story, it caused me to start looking for other incorrect details. That was probably not the intention of the director.
Your characters must live, work and play somewhere. If you never mention the setting, your readers will not be able to get a feel for the characters. On the contrary, if you talk about every little detail that comes to mind, your book will be so logged down, no one will read it. You need to provide just enough detail to make the story and characters come alive for your readers.
If your story takes place in a real setting and the details are either critical to the story, or it’s well known by many people, ensure you fact check enough to not make glaring mistakes, such as the location of landmarks. If you have personal knowledge of the location, feel free to add smaller details that locals might appreciate so long as they don’t distract from your story. Most readers will forgive minor mistakes, but it’s best to err on the side of caution.
If you are creating your own world, your readers don’t have any frame of reference, so you’ve got a free rein. Of course, you also have the responsibility to write just enough detail so your readers can visualize your world. If your character’s local environment isn’t clear enough, they might not feel real enough for readers to care about them.
Most stories take place in a fictionalized local within the real world. There is nothing wrong with a local setting that doesn’t exist provided it feels real enough for the region. Keep the travel times from landmark locations consistent with the real world, but feel free to add neighborhoods and even towns that fit in. If your mythical small town in Nevada has a forested area within walking distance, you’ve got believability issues. But, provided you don’t use real street names, a fully realized fictional neighborhood around a fictionalized hospital in New York is acceptable to most readers.
Writers today have the advantage of easy fact checking, but so do readers. If your characters live in an environment that feels real enough, your readers won’t be tempted to question the imaginary details you provide.