I want to develop code to ensure that all of the data in a data file is
loaded correctly into my MySQL database. I can write a fairly simple
perl script to whether or not the data feed received is correct, or at
least that the records received are complete and do not contain
invalid characters. As far as I can tell, we can not know that all
records that are supposed to be received have been received, but if
there are supposed to be 10 fields in a record, we can know whether or
not the last record received has 10 complete fields.

As I said, I can write a perl script that can check the completeness of
a record and that all data in it is valid for the type expected.
However, I am not sure how I could determine whether or not all of the
data in my data files are loaded properly.

What do you regard as the best practices for ensuring the quality of
the data in your databases?

Surely I don't have to just assume that the data was loaded correctly
if my script that loads the data runs to completion.

Does anyone have any ideas about what could go awry and how one would
check for data corruption that might result?

Thanks,

Ted