What A Load Of Business Data

Auntie Pat Tern thinks Superman's x-ray vision is stupid because "how would he ever know where he's supposed to look and where he should not bother focusing?" That got me wondering about the data and tests in my Salesforce org.  Even clicks-not-code developers should have automated tests that validate configuration changes with existing and expected data. So what's the best way to know which data those automated tests should use?

Some Salesforce developers like to write automated tests with "SeeAllData=True", an outdated and bad practice. A test that can see all data is not the same as a test that is smart enough to see the data that needs testing. For example, a good test will test positive scenarios, negative scenarios and extremes. In other words:

• What if the data is exactly what we planned for -- records with all the right data in all the right places?
• What if the data falls outside of expected norms -- records with missing fields or invalid data, for example?
• What if the data is coming in from a data import operation or an integration and many records need to be processed?

Admins can follow a few simple steps to create automated tests to run against their configuration changes, and their efforts can be used by developers for better unit tests as well. Simply create sample data representative of the expected inputs. Then use code like the following to load that data in a unit test:


Note that you will want to use data that corresponds to the configuration changes that you are testing. In this example, configuration changes and business process automation around Account, Contact and Opportunity record creation will be tested. You can create a similar test class for other related objects, including custom objects.

This code can be run as a unit test by itself to validate configuration changes, or it can be called by other test code to set up data for more complex unit testing related to other code in the org.

Next week, I will look into some of the errors that might occur when you create CSV files of sample data and how to avoid those errors.

But for now, Auntie Pat Tern has a point, who wants Superman looking at just any old thing with his x-ray vision when he ought to be focusing on information that's most helpful.

Enough Is As Good As A Feast

Auntie Pat Tern loves Sir Thomas Mallory and Mary Poppins, and quotes both when she reminds us "enough is as good as a feast", especially when Tim Toady's eyes are bigger than his stomach at the buffet. So I thought I would look at Salesforce storage limits to see if we have a feast in our Enterprise Edition org (which is the entry-level org for many businesses and the basis for the nonprofit license grant, even those with the nonprofit starter pack pre-installed).

Salesforce currently allocates 20MB of data storage per user in every EE org. But wait, there's more. Every org starts with a minimum of 1GB of data storage. So whether you are a nonprofit with a grant of 10 free licenses or a small business with 50 licenses, you have a minimum of 1GB for data.

More than enough is too much.

But how many records is that? Salesforce allocates 2KB for most records. Articles require 4KB, Campaigns require a whopping 8KB (4 times the size of most other records). That means a basic Salesforce org (Enterprise Edition or better) will have data storage capacity for about 500,000 records (excluding Campaigns and Articles). If you have 100,000 Accounts and 400,000 Contacts for 1-50 users, you are going to need more storage.

If you are using Person Accounts, or if you create a unique Account for each Contact, 250,000 individuals would result in 500,000 records since each individual would require both an Account and Contact record.

Custom objects behave the same as typical standard objects taking 2KB per record, regardless of the number or type of fields associated with those records. Even if you use a lot of rich text fields with large image files, a custom object record still requires only 2KB of data storage. The trick with rich text fields is that they actually are stored as files and so impact file storage rather than data storage.

Consider your storage needs carefully when you create sandboxes. Partial Copy sandboxes currently provide 5GB of data storage, that's more than you have in your production org if you have 50 or fewer users! But they don't offer much in terms of file storage. You may need a Full Copy sandbox for the convenience of copying all of your data at once and accommodating larger amounts of file storage and files from rich text fields.

But for standard data needs, use these basic formulas to calculate your data needs in GB:

( # of records X 2KB ) X 1/1024 X 1/1024

or

( # of Campaigns X 8KB ) X 1/1024 X 1/1024

or

( # of Articles X 4KB ) X 1/1024 X 1/1024

In the first formula, we multiply the number of records you have times 2KB, the data storage typically needed for records to get the required number of kilobytes of storage. Then we multiply by 1/1024 to convert from kilobytes to megabytes and again to convert from megabytes to gigabytes.

In the second formula, we multiple the number of Campaign records times 8KB because they consume a lot of storage compared to typical standard and custom objects. Then we do the same calculations to convert from KB to GB as described above.

The third formula is for calculating the storage needs of Articles since their records require 4KB rather than the standard 2KB. Otherwise it works like the first two, with the result expressed in gigabytes.

Keep in mind that Person Accounts and 1x1 Contact to Account data models will create a Contact and an Account record, so every individual is represented by two records rather than just one.

Salesforce offers options for purchasing additional storage space. You can also upgrade to Performance, Unlimited or similar Editions that offer six times as much data storage per user compared to Enterprise Edition (orgs with fewer than 10 licenses may not see the benefits of the additional storage).

But, as Auntie Pat Tern would say, it is good to be grateful for what you have and know when you've got enough for your share. Salesforce makes it easy to calculate your needs.

More Better Bots

You know Auntie Pat Tern loves Jeopardy!, the tv show with all the answers, but we have an old friend named Eliza who believes we can find all the answers we need by asking ourselves the right questions.  My developers have been asking some of the right questions since their recent trip to the TrailheaDX conference (a conference for Salesforce Developers, similar to past conferences in 2007 and 2010).

First of all, the developers asked what we should do about all of our existing Visualforce pages now that users are craving Lightning Experience.  "What can we do to give them that look and feel without a wholesale rewrite of our UI?" they asked.  Their questions naturally led them to the answer that we should start simply with applying the Lightning Design System to what we have in Visualforce, as a first step.

Next, the developers asked what was going on with Salesforce and AI.  "What's the deal with bots?" they wondered.  While they mulled this over, I reached out to Eliza to see what she had to say:

> Hello. My name is Eliza.  How may I help you?
* Eliza, what's the difference between AI and machine learning?
> Ahh.  Please continue...
* Really, I have three issues: Artificial Intelligence, Machine Learning and Natural Language Processing.
> What does that suggest to you?
* Like Steve Wozniak said at TrailheaDX, Artificial Intelligence is nothing without Machine Learning.  Natural Language Processing benefits from the ability to learn better responses.
> I'm not sure I understand you fully.
* Well, GA Tech has done some cool stuff with bots.  One learned to answer frequently asked questions for students.
> Can you elaborate on that?
* For a class that prompted over 10,000 online questions per semester, the professor used Machine Learning and trained a bot using questions and answers from previous semesters.  The bot was set to respond to those familiar, frequently asked questions and to only respond to new questions if it calculated a 97 percent or higher confidence level for its answer.
> What does that suggest to you?
* It suggests that I can blend Natural Language Processing and Machine Learning to provide automated, interactive responses to FAQs.
> Say, do you have any psychological problems?
* Um, I do sometimes chat with bots.

Having a bot that can help customer service agents provide the best responses to Cases would help our service team reply quickly and consistently.  We also might want a bot to help provide strategies for the sales team, following the department's best practices. Bots, with the ability to learn from our org's data and best practices can also help us guide new employees and reduce their frustration.

Auntie Pat Tern doesn't really like the term "Artificial Intelligence" because she believes that using code to process our human questions and requests is genuine intelligence.  And our old friend Eliza is curious to know how you feel about that.

SeeAllData = Fail

According to Auntie Pat Tern, it takes 21 days to break a bad habit, but Salesforce developers have had about that many API versions to break the bad habit of writing unit tests with "SeeAllData=true".

In most cases, this clause alone can render tests completely useless. Unit tests should prove that code functions according to specifications by asserting proper database manipulations--create, read, update, delete operations--as well as Visualforce navigation.

I looked at unit tests in my org and found this example of a test that failed to achieve its purpose:

This test should be asserting that the data for ordering samples can be created with just a contact and sample type defined.  Unfortunately the test relies on existing, live, org data rather than test 'silo' data because of "SeeAllData=true" in the first line. There are easy methods for unit tests to create their own test data without relying on live org data.

We encountered the following problems because of using "SeeAllData=true":

• It required us to maintain test data among our real data.
• When the test data was changed during data cleanup (in one case, Pat Tern's Account was deleted), the tests failed even though functionality was unchanged.
• Tests were not reliable between Sandboxes and Production orgs due to data differences rather than actual functionality.
• Apex Hammer Tests may not have been automatically monitored in our org for each Salesforce release since Hammer Tests are blind to live org data.

In the rare case where a specific piece of data may be required for your code to behave, consider using custom metadata types instead of Salesforce objects. Trailhead can help you learn more about how they allow you to move metadata and records between orgs and test functionality without needing to see all data in the org.

Pay Down That Technical Debt

Auntie Pat Tern says we should pay with cash, or at least pay our credit cards off every month because "debt," she says, "is like fresh fish, it seems great at first but gets old fast and then it really stinks."

It's Hammer Time!

So I looked at my Salesforce org to assess the level of technical debt we had accrued and to plan how we would start paying down that debt.

Sometimes technical debt comprises the shortcuts or mistakes that no one has time to correct when a project needs to be completed. Technical debt also develops over time.  As businesses mature and change, their technical solutions can fall behind and create technical debt.

A Salesforce org can be especially prone to this since Salesforce offers a wealth of new features for all orgs three times each year and not all companies make an effort to rewrite their technical solutions based on these new features. Luckily, Salesforce offers some tools to help us assess some of the technical debt associated with our code.

First, I checked our Hammer Test Status.  The data silo gauge revealed that we still had old (and sadly a few new) unit tests that were using "seealldata=true", which we needed to update.  It also revealed a couple of tests that were failing and so needed some attention.

Next I checked the API version on our Apex Classes.  Any class that is 10 or more versions behind the current API version needed a review, both on the code side and on the process side.

The third step for me was reviewing our org's documentation for outdated information on our code.

My org review also included configuration, managed packages and basic processes, as well.

We may not be able to pay down all of our technical debt right away, but a monthly review and reminder of how it accumulates will help us develop better in the future.  And as Auntie Pat Tern says, "It's not just pay me now or pay me later, you know.  It's pay me now or pay me later with additional compounded interest!"

Putting Governor Limits To The Test

My Auntie Pat Tern is pretty accepting of cousin Tim Toady's behavior. "He's a teenager, after all, there's no better time for him to test his limits," she explained. So I decided to ask my developers to test their limits in Salesforce.

One Master object with 13 Detail objects, 
some of which are Masters in other relationships as well.

I wanted them to see for themselves how governor limits benefit the overall performance of the code they write. And I wanted them to experiment with ways to push those limits by trying to break things.

They built Processes, wrote triggers, and configured some unwieldy objects that I would never want to see in production all in an effort to push good performance to the very edge of being bad.

Some limits include object relationships.

A couple of these experiments proved that what they understood about limits was untrue. In one example, when it comes to Master-Detail relationships on custom objects, the documentation described a limit of 2^3. Firstly, 2^3 does not equal 8 here. Instead, it indicates that an object can have two M-D relationships and those can be three levels deep.

Take the example Parent <-- Child <-- GrandChild <-- GreatGrandChild where all relationships are Master <-- Detail. Some of the limits on this relationship structure are as follows:

• Parent cannot have a new Master (eg GrandParent) because of the limit on how deep the relationship levels can be. 
• GreatGrandChild does not show up as available to be a Master in other relationships because we are limited to three levels deep.
• Child cannot have a new Master because it already has two M <-- D relationships (even though only one of those points to a Master).
• Child can have new Details, that is, new GrandChild-level objects can be created as Details for Child even though Child already has two M <-- D relationships.
• Many Child-level objects can be created as Details for Parent (we stopped at over 50). 
• A Child-level object cannot be used as a junction object between records of a single Parent-level object. M <-- D relationships cannot be immediately self referencing like that.
• GrandChild-level and GreatGrandChild-level objects can have the same Master object as their Master, eg. GrandChild can point to Child and Parent as Masters even when Child already points to Parent as its Master. We daisy-chained six objects this way before hitting limits on the depth of the relationships.
• Child-level objects with two relationships pointing to Master objects cannot be Masters to new GrandChild-level objects. An object can have two Masters or it can have one Master and many Detail relationships or it can have no Master and many Detail relationships.

It was a fun exercise and demonstrated how limits benefit performance and how hard some of them can be to break. It gave the developers a chance to challenge their assumptions, be creative and gain a better understanding of the implications of limits when it comes to writing better code.

Secretly, Auntie Pattern believes that testing limits can help us appreciate why limits are important, but she wouldn't tell Tim Toady that.

Painless Removal Of Boat Anchors

I asked my Auntie Pat Tern why she has a tattoo of a boat anchor on her forearm and she said, "Boat anchors symbolize hope, but I'm considering getting this one removed." So she didn't have it to enhance her resemblance to Popeye like cousin Tim Toady always says.

It may have been hope that also inspired the boat anchor I found in the code in my org. In technical circles, some people only think of boat anchors as that outside technology they got stuck with because their boss bought it without conducting a technical review. But if you aren't constantly conducting a technical review of your own code, you can get stuck with boat anchors there as well.

In my org, I found something like the following:

All code that had once handled business logic was commented out to the point where the code literally did nothing but returned a null value. I think this boat anchor represented the developer's hope that the original code might somehow be useful later. If that's the hope, the code can be stored outside of our production org.

To remove it, we just needed to correct any references and delete the class and its related unit test. Salesforce offers multiple ways to accomplish this using the IDE, the developer Workbench or the Migration Tool, all of which are much less painful than Auntie Pat Tern's tattoo removal.

With Workbench, for example, you can simply mark a class or trigger as deleted or as inactive to remove it from production using the following steps after downloading the class or trigger files:

1. Edit the XML meta file for the class or trigger to change its status from <status>Active</status> to <status>Deleted</status> (for classes) or <status>Inactive</status> (for triggers)
2. Put the both the .XML file and class or trigger file into a folder named "classes".
3. Create a package.xml file to list the class or trigger to be deleted or made inactive.
4. Place both the package.xml file and the classes folder into another folder inside a zip file.
5. Specify this zip file after you select Migration>Deploy in Workbench.

A great way to generate a package.xml file to get you started is to use Setup>Create>Packages in Salesforce and add the classes or triggers that need to be deactivated or deleted. This package can then be specified for download in Workbench using Migration>Retrieve. The zip file will need to be changed as described above to deploy. (For step-by-step instructions, click here.)

Like Auntie Pat Tern, be on the lookout for boat anchors and consider removing them when they pop up during your org's ongoing technical review.