When you click Go Back, the program cancels the last action you have taken to modify your segment.
For example, if you have added all the contacts with the business relation Vendor, the program automatically removes them again from the segment.