In most cases, this clause alone can render tests completely useless. Unit tests should prove that code functions according to specifications by asserting proper database manipulations--create, read, update, delete operations--as well as Visualforce navigation.
I looked at unit tests in my org and found this example of a test that failed to achieve its purpose:
This test should be asserting that the data for ordering samples can be created with just a contact and sample type defined. Unfortunately the test relies on existing, live, org data rather than test 'silo' data because of "SeeAllData=true" in the first line. There are easy methods for unit tests to create their own test data without relying on live org data.
We encountered the following problems because of using "SeeAllData=true":
- It required us to maintain test data among our real data.
- When the test data was changed during data cleanup (in one case, Pat Tern's Account was deleted), the tests failed even though functionality was unchanged.
- Tests were not reliable between Sandboxes and Production orgs due to data differences rather than actual functionality.
- Apex Hammer Tests may not have been automatically monitored in our org for each Salesforce release since Hammer Tests are blind to live org data.