Nothing. But gypsies are largely harmless, and they were nomadic. Meaning they were rather rare to see and isolated for centuries. They do however low-level physical labor sometimes, and are notorious open market vendors who sell stuff cheap... which probably was stolen.
jews i get because money lending and shit
but why gypsies? what did gypsies had to offer to kings and kingdoms that made them tolerate their existence?