You’re probably better off researching 301 redirects for the pages that Google returns in the search results. This way if anyone does a search, they will end up at the correct new page. It will also help you keep your page rank on the pages you’ve essentially moved.
There’s lots of info on 301 page redirects both on here and on Google. Just do a search.
I agree with Don @dlh,
If you are showing up on any SERP (search engine results page), it’s a good thing!
When a page moves it’s best practice to redirect to the new location. It benefits more than just Search engines. Folks bookmark pages and pages may get linked to from blogs, forums or other sites.
When you go to rebuild a website yours or someone else’s it a good to add a step to the workflow to map each page of the old site to the equivalent content on the new site. There isn’t always a one for one match but try to decide where someone should “land” if no matter where they have the old link from. It helps with SERP ranking and provides a better user experience.