I think the 6 months is only while you are waiting for the application to be processed . Once you have residency & are in receipt of the tarjeta it is no more than two years in a five year period.
Yes , here it is from the EU rules;
" The same rule applies to family members who are not nationals of a Member State and who have lived with a Union citizen for five years. The right of permanent residence is lost only in the event of more than two successive years' absence from the host Member State."
From here , scroll down to " Right of Permanent Residence "
http://europa.eu/legislation_summaries/internal_market/living_and_working_in_the_internal_market/l33152_en.htm