Both the sins mentioned in this Hadith are such that if a Muslim thinks them lawful and still commits them he will become a Kafir. To defame someone's lineage means to disgrace or humiliate somebody by saying to him or to her: "Your father belongs to such and such profession" or "Your mother is such and such / so-and-so" or "You are a weaver, blacksmith, laundrer, cobbler, etc."
Mourning and wailing means expressing qualities of a deceased person through weeping, crying and lamenting loudly. Imam An-Nawawi said that such qualities are attributes and acts of the disbelievers and practices of the pre-Islamic period, or Jahiliyyah.