July 17, 2017 by agarciaodeian

Salesforce BigObjects

What is a BigObject?

Two years ago I started working with Big Objects, a Salesforce pilot that looked very interesting for me. And now that it seems that General Available time is closer, I would like to write about them.

Please, take into account that a Salesforce pilot could be discarded like Data Pipeline that it would not be GA at the end.

Big Objects is a new type of objects that Salesforce provides. Maybe the most important thing I can tell is that they do not count against the storage limit. Well, you could say that External Objects already do.

You can learn more about External Objects here.

Yes, you are right. But this time, data lives in your organisation instead of in an external storage and this means that you can work with them like with Custom / Standard objects, but if you go to Salesforce Storage screen, you would not find them.

Big Objects let you store and manage large amounts of data.

Use Case

But before starting with the explanation, let’s talk about the Use Case.

As a company that creates software, we have stories in order to develop new functionality. Before delivering them, we need to pass a code review, so that, we ensure the best to our end users.

Captura de pantalla 2017-07-07 a las 12.21.47

But how much storage should I use to keep code review records in our Production org? Should I remove them? If so, what about auditing?

BigObjects is your solution, so that, you can move old records to a new Code Review History BigObject record. Let’s focus for now on this new History object.

How to Create a BigObject?

Unfortunately you cannot do it in a declarative way. You need to define them via Metadata API and then do a deployment via cmd or workbench.

Before doing it, take into account:

Indexes are required and in order to include them, you need to be on API 39.0 onwards.

Once you deploy one, you cannot make any modification like amend a field type, change label, etc. However in a declarative way you can do small changes like the label and API name.

<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
  <deploymentStatus>Deployed</deploymentStatus>
  <fields>
    <fullName>CanBeMerged__c</fullName>
      <label>Can Be Merged</label>
      <length>80</length>
      <type>Text</type>
    </fields> 
  <fields>
  <fullName>CodeReviewDate__c</fullName>
    <label>Code Review Date</label>
    <type>DateTime</type>
    <required>true</required>
  </fields>
  <fields>
    <fullName>Comments__c</fullName>
    <label>Comments</label>
    <length>255</length>
    <type>Text</type>
  </fields>
  <fields>
    <fullName>Employee__c</fullName>
    <label>Employee</label>
    <referenceTo>Employee__c</referenceTo>
    <relationshipName>Employees</relationshipName>
    <required>true</required>
    <type>Lookup</type>
  </fields>
  <fields>
    <fullName>Score__c</fullName>
    <label>Score</label>
    <precision>1</precision>
    <scale>0</scale>
    <type>Number</type>
  </fields>
  <fields>
    <fullName>Story__c</fullName>
    <label>Story__c</label>
    <referenceTo>Story__c</referenceTo>
    <relationshipName>Stories</relationshipName>
    <required>true</required>
    <type>Lookup</type>
  </fields>
  <indexes>
    <type>PRIMARY</type> 
    <fullName>Test1CodeReviewPK</fullName>
    <fields>
      <name>Story__c</name>
      <sortDirection>DESC</sortDirection> 
    </fields>
    <fields>
      <name>Employee__c</name>
      <sortDirection>DESC</sortDirection>
    </fields>
    <fields>
      <name>CodeReviewDate__c</name>
      <sortDirection>DESC</sortDirection>
    </fields>
  </indexes>
  <label>Test1 Code Review History</label>
  <pluralLabel>Test1 Code Reviews History</pluralLabel>
</CustomObject>

Once you deploy it into your organization, you would find something like this under BigObject entry:

I decided to name is as Test1 Code Review History because once it is created I cannot modify it, and I didn’t want to use Code Review History label for now.

Captura de pantalla 2017-07-14 a las 13.03.54.png

What can we highlight?

The API name ends with __b instead of __c or __x like custom and external objects.
We will not find Standard fields.
We can create as many custom fields as we want but right now it only supports these field types: DateTime, Lookup, Number, Text and Long TextArea. What about the rest? Try to be creative. For instance a Checkbox can be converted into number or text.
You will not find any other section like PageLayout or Buttons for now.
You will not find Triggers section. It is not allowed for now.
You will not be allowed to create a custom tab related to this object. That is related to previous point and the PageLayout. To visualise Big Object records, you need to create a visualforce page or a lightning component.
You can determine its CRUD and FLS via Profile or Permission Sets during the deploy process or in a declarative way. However for now BigObjects can be only created and read.

Captura de pantalla 2017-07-14 a las 13.44.36

How Can I Create Records?

There are different ways to create BigObject record, like using a csv file, use APIs like Bulk API or even Async SOQL, another Pilot I will talk about in a future entry.

In any case, here, I will focus on Apex.

As I mentioned above, you can only Read or Create records, and nowadays, you cannot use simple DML operations, there is a new one: database.insertImmediate(record)

Test1CodeReview__b cr = new Test1CodeReview__b();
cr.CanBeMerged__c = 'False';
cr.CodeReviewDate__c = System.today();
cr.Comments__c = 'I found a SOQL inside of a loop.';
cr.Employee__c = 'a6j24000000fxSL';
cr.Score__c = 0; 
cr.Story__c = 'a6k24000000k9bN';
database.insertImmediate(cr);

But something pretty cool that it was not available at the beginning, it is the fact we can use this method, to also update records. Actually it works like upsert. If I make a call where the record has a primary key that doesn’t exist, it would create a new record as we can see in the below image.

Captura de pantalla 2017-07-14 a las 14.28.04

However if the value already is in the platform, it will make a modification.

Test1CodeReview__b cr = new Test1CodeReview__b();
cr.CanBeMerged__c = 'False';
cr.CodeReviewDate__c = System.today();
cr.Comments__c = 'I found a SOQL inside of a loop.';
cr.Employee__c = 'a6j24000000fxSL';
cr.Score__c = 1; //Change to 1
cr.Story__c = 'a6k24000000k9bN';
database.insertImmediate(cr);

If I come back to the original use case, I would like to move CodeReview__c records to Test1CodeReviewHistory__b records. In order to do that I only need to retrieve the record I need and create a new BigObject entry.

CodeReview__c cr = [Select Id, CanBeMerged__c, 
                           CodeReviewDate__c, Comments__c, 
                           Employee__c, Score__c, Story__c 
                    From CodeReview__c 
                    Limit 1];

String canBeMerged = cr.CanBeMerged__c == true ? 'True' : 'False';

Test1CodeReview__b crh = new Test1CodeReview__b();
crh.CanBeMerged__c = canBeMerged;
crh.CodeReviewDate__c = cr.CodeReviewDate__c;
crh.Comments__c = cr.Comments__c;
crh.Employee__c = cr.Employee__c;
crh.Score__c = cr.Score__c; 
crh.Story__c = cr.Story__c;
database.insertImmediate(crh);

How Can I Visualise Records?

As we have already mentioned, we cannot create a custom tab related to a BigObject and show all records. However we can query BigObjects, so what about if we create a custom page for that?

First of all we would need a controller. Really simple one. We have a method to retrieve all records and a get and set to show in a list what we retrieve.

public with sharing class CodeReviewController
{
   private static List<Test1CodeReview__b> codeReviewHistoryList; 
   
   public CodeReviewController()
   {
      setCodeReviewHistoryList(calculateCodeReviewHistoryList());
   } 

   public List<Test1CodeReview__b> calculateCodeReviewHistoryList()
   {
      return [SELECT Id, CanBeMerged__c, CodeReviewDate__c, 
                     Comments__c, Employee__c, Score__c, 
                     Story__c                
               FROM Test1CodeReview__b];
   }

   public static List<Test1CodeReview__b> getCodeReviewHistoryList()
   {
      return codeReviewHistoryList;
   }

   public static void setCodeReviewHistoryList(List<Test1CodeReview__b> value)
   {
      codeReviewHistoryList = value;
   }
}

Secondly, we have the visualforce page. Something to highlight is that right now, the standardController attribute doesn’t work, so we can only create a page with a customController.

<apex:page showHeader="true" sidebar="true" controller="CodeReviewController">
   <apex:form>
      <apex:pageBlock title="Code Review History Records">
         <apex:pageBlockSection title="Code Review History" columns="1" collapsible="false">
	    <apex:pageBlockTable value="{!codeReviewHistoryList}" var="crHistory">
               <apex:column headerValue="Ready To Merge?">
	          <apex:outputField value="{!crHistory.CanBeMerged__c}"/>
               </apex:column>
	       <apex:column headerValue="Score">
	          <apex:outputField value="{!crHistory.Score__c}"/>
	       </apex:column>
               <apex:column headerValue="Comments">
	          <apex:outputField value="{!crHistory.Comments__c}"/>
               </apex:column>
               <apex:column headerValue="Story">
	          <apex:outputField value="{!crHistory.Story__c}"/>
               </apex:column>
               <apex:column headerValue="Reviewer">
                  <apex:outputField value="{!crHistory.Employee__c}"/>
               </apex:column>
               <apex:column headerValue="Code Review Date">
                  <apex:outputField value="{!crHistory.CodeReviewDate__c}"/>
               </apex:column>
            </apex:pageBlockTable>
         </apex:pageBlockSection>
      </apex:pageBlock>
   </apex:form>
</apex:page>

And the result is this one that includes the record created from scratch and the one that is retrieved from CodeReview__c object.

Captura de pantalla 2017-07-17 a las 9.14.52

Summary

I would like to share this table that summarise BigObjects and compare with Custom Objects. But before, I would like to also highlight that this new object type can be included in a package. You would find under CustomObject section:

Captura de pantalla 2017-07-17 a las 9.16.06

Summary Table:

Feature	Custom Object	Big Object
Creation	Manual Metadata	Metadata
API name	myObject__c	myObject__b
Track Activities Track Field History, etc.	Options Available	Options No Available
Field Types	All	Text ; Date/Time ; Lookup Number ; Long Text Area
Able to edit fields	Yes	Yes (with restrictions)
Able to delete fields	Yes	No
Triggers; Field Sets; etc	Options Available	Options No Available
Reports	Yes	No
How to Populate records	All	CSV file API (Bulk, SOAP) Apex Async SOQL
Can I amend a record?	Yes	Yes (with restrictions)
Can I see data creating a Tab	Yes	No
For free?	Yes	No — Talk with Salesforce
Storage	It count against storage	It doesn’t count against storage

38 thoughts on “Salesforce BigObjects”

Rajesh Ramachandran | July 29, 2017 at 2:09 am

thanks, this is very insightful

LikeLiked by 1 person

Reply
sfdeveloper | August 3, 2017 at 11:16 am

Thank you Augustina for the great article.
How do we enable big objects in dev Edition org. Do we need to create a case with salesforce. where do we need to do this

LikeLike

Reply
- agarciaodeian | October 24, 2017 at 4:33 pm
  
  Sorry for the late response. Yes, actually you had to ask to have the pilot activated via a case. But now, in Winter ’18 the feature is General Available
  
  LikeLike
  
  Reply
Renato Oliveira | October 10, 2017 at 7:47 pm

That upsert doesn’t make sense to me. How can the platform know that you are updating the information instead of inserting a new one, if you are not providing an Id? To me, a upsert call will always insert a new record if you don’t pass an Id.

LikeLike

Reply
- agarciaodeian | October 24, 2017 at 4:32 pm
  
  Sorry for the late response. Not sure if you realized about indexed fields. They work as primary key. So when you try to insert a record, it looks the PK and if it finds a record with same values as PK, it update the record.
  
  LikeLike
  
  Reply
Esteve Graells | October 24, 2017 at 3:08 pm

Muchas gracias Agustina, me ha parecido una entrada muy completa y muy interesante. Es una característica que en mi opinión, en cuanto deje de tener las limitaciones actuales, puede ser un bombazo.
Nuevamente gracias,
Esteve

LikeLike

Reply
Esteve Graells | October 24, 2017 at 3:23 pm

Perdona Agustina, respecto a las limitaciones que comentabas en tu artículo, añadiría que ciertos operadores en las SOQL no están permitidos, te paso la frase exacta de la documentación: The !=, LIKE, NOT IN, EXCLUDES, and INCLUDES operators are not valid in any query.

LikeLike

Reply
- agarciaodeian | October 24, 2017 at 4:28 pm
  
  Gracias Esteve. Es muy gratificante tener feedback y si además añade información para otros es genial.
  Después de Dreamforce ’17 publicaré otro sobre Async SOQL. Espero que te guste también.
  
  LikeLike
  
  Reply
Pingback: Big Objects: Contenedores de Datos – un quiero y quizás puedo de almacenamiento masivo en Salesforce – Artículos sobre Salesforce
Prashant | November 17, 2017 at 10:15 pm

Good Introduction to Big Data..It’s really helpful.
Could you please tell me how can we use the big object to Track Field History for custom/standard sobject.
I have an idea that we can create a big object and relate to a custom or standard object and then create an after update trigger on that object that will insert the field changes into the custom big object…but not sure that is a good solution.

LikeLiked by 1 person

Reply
- agarciaodeian | November 18, 2017 at 1:27 pm
  
  Thanks!! Happy it was useful for you. Actually there is a Standard Big Object called
  FieldHistoryArchive is a standard big object that stores data as part of the Field Audit Trail
  product. Not sure if this can help. Otherwise, yes it make sense your idea about the after update trigger.
  
  LikeLike
  
  Reply
  - agarciaodeian | November 25, 2017 at 10:22 am
    
    You can find an example on How can I create records? section. That piece of code create records on a simple method. However I’m not so sure that you can use it inside of a custom object trigger. I have asked product manager and waiting for his response.
    
    LikeLike
Sudipta Halder | November 20, 2017 at 7:07 pm

Good article on big object.
Could you please let me know whether we can insert any records to bigobject from any trigger(standard/Custom ).

As i have a requirement to store changes history(any field value changes) of opportunity split to a big object. where we are planning to use after insert/update/delete trigger on opportunity split to save all the information on abigobject.

So i am not sure whether is it possible or not. Please help.

LikeLike

Reply
- agarciaodeian | November 21, 2017 at 8:24 am
  
  Thanks Sudipta. Actually you can create records via Database.InsertInmediate. You can find more information on How Can I Create Records? section.
  
  To be honest, I have not tried in a trigger. I guess there is no issue. At the end it is apex code, but keep in mind bulkification
  
  LikeLike
  
  Reply
Pingback: 10 thoughts about Big Objects – My Salesforce adventure
Vasant | November 21, 2017 at 9:53 pm

Error in inserting records in Big objects via Apex

I have used Database.insertImmediate(objOppSplitHistories); method to insert/update records into OppSplitHistories (my custom big object).
Here I have used the above method in apex class and called that method from oppsplit trigger to pass the old map/new map info for saving opp splits changes. But that is throwing the below error. I have observed if I run the same piece of code in ‘Execute Anonymous Window’ then data is being inserted in to custom big object but despite using insertImmediate method it is not working through trigger.

Error: Update failed. First exception on row 0 with id 049c0000002JaopAAC; first error: UNKNOWN_EXCEPTION, EXTERNAL_OBJECT_UNSUPPORTED_EXCEPTION: Apex triggers can’t reference the Big Object BigObjectCounter: []

It seems we cannot populate any bigobject through trigger . The considerations only says that we cannot write Triggers on Big object but it does not say that we should not call the InsertImmediate method from a Trigger (on a Custom object). Its not clear. Any help or clarity will be much appreciated.

LikeLike

Reply
- agarciaodeian | November 22, 2017 at 1:46 pm
  
  Thanks for sharing the information. Really useful for others too. I will double check and back to you.
  
  LikeLiked by 1 person
  
  Reply
Prashant | November 25, 2017 at 7:25 am

Vasant -Could you please share the whole code..
I have used Database.insertImmediate in the trigger on custom object..

LikeLike

Reply
- agarciaodeian | November 25, 2017 at 10:26 am
  
  You can find an example on How can I create records? section. That piece of code create records on a simple method. However I’m not so sure that you can use it inside of a custom object trigger. I have asked product manager and waiting for his response.
  
  LikeLike
  
  Reply
agarciaodeian | November 28, 2017 at 8:44 am

I have talked with PM and he confirmed that it is not posible to create a big object record inside a trigger. However a workaround could be to put the insertImmediate() call into a queueable method – in this way it separates the operation from the ongoing trigger’s transaction.

LikeLike

Reply
Pingback: Big Object field types workarounds – My Salesforce adventure
Pingback: Process billions of records with Async SOQL – Agustina odeian
Sourav Nandy | January 3, 2018 at 10:51 am

Can i use the lookup feild on Big objects and map it to a custom object?

LikeLike

Reply
- agarciaodeian | January 8, 2018 at 8:40 am
  
  Yes, you can, but only to Standard or Custom objects. You cannot create a lookup to a big object for now.
  
  LikeLike
  
  Reply
Hybin | February 7, 2018 at 12:53 pm

Thankyou for putting this together. I have implemented everything just got stuck with Querying records inside the Big Objecheduled Job, however I dont seem to be able to search or query records once they are moved to the Big Object. Any chance this is possible cos I cant use LIKE for searching. If anyone has come up with a workaround it would be great to know… Thanks

LikeLike

Reply
- agarciaodeian | February 16, 2018 at 10:10 am
  
  Hi Hybin. For sure you can query them. I’m not so sure about searches as I didn’t try by myself, but looking at this page I would say they do not support yet. https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/big_object_querying.htm
  
  LikeLike
  
  Reply
  - Hybin | February 26, 2018 at 7:01 am
    
    Thanks I got the Search part working now have another problem we started of with this so that we could archive records but the storage for 1 big object seems to be 1 Million Records(Correct me if I am wrong). If we have multiple Big Objects how would we know when they reach their Limit? For example I have 2 big objects but the Storage still show 1 Million
    
    I wish I could share a screenshot… I am talking about the Setup>Storage> and then if you look at Big Object section in Storage. I will be checking this with Salesforce as well but if you have any info on this please let me know.
    
    LikeLiked by 1 person
Pingback: BigObjects and Triggers – Agustina odeian
agarciaodeian | February 27, 2018 at 6:46 am

Hi Hybin

Yes, I know what you mean. If you go to Storage Usage page, the top section is the total storage in the org, where you can see the 1 Million records, the total number of records you have used as well as the %.

If you scroll down there is another section, “Current Big Objects Storage Usage” where you can see the list of big objects you have, and for each row, the number of records you have used. For instance I have 4 big objects. First one shows me 5 records, 2nd, 10 records and so on.

Regarding the Million of records, that is the free amount SF provides, so if you need more, just ask Salesforce.

BTW, how did you solve the SOSL issue? Could you share the code here?

LikeLike

Reply
Armando | April 10, 2018 at 8:34 am

Hi Can get a test class for this example??

LikeLike

Reply
- agarciaodeian | April 14, 2018 at 7:08 pm
  
  TBH I didn’t create it, but I will work on it and back to you.
  
  LikeLike
  
  Reply
HITENDRA SABHADIYA | August 14, 2018 at 5:11 am

How to write Test class for for Apex which is manipulating Big object?

LikeLike

Reply
- agarciaodeian | September 12, 2018 at 7:46 am
  
  Sorry for the big delay but I have been very busy and not able to take a look at that. Once I do it, I will replay these comments.
  
  LikeLike
  
  Reply
Anil | September 21, 2018 at 1:51 pm

NIce article with more information. Thank you so much

LikeLiked by 1 person

Reply
Ramakrishna | November 29, 2018 at 2:32 pm

Hi agarciaodeian ,

I am also one of them trying to get test coverage for big objects

Thanks to your page I was able to learn some tips about this and would be very happy if you could help me with getting coverage on this :

webservice static void myMethod()
{
list jd = [SELECT Backup_Taken__c,Name,Related_Account__c,Closing_Balance__c FROM Transactions__c WHERE Backup_Taken__c = False ];
List testtrans = New List();
list jr = New list();
if(jd != null && !jd.isEmpty()){
for(Transactions__c cr:jd){
All_Archived_Transactions__B crh = New All_Archived_Transactions__B();
crh.Related_Account__c = cr.Related_Account__c;
crh.Closing_Balance__c = cr.Closing_Balance__c;
crh.Transaction_No__c = integer.valueof(cr.Name);
cr.Backup_Taken__c= true;
testtrans.add(crh);
jr.add(cr);
}
database.insertImmediate(testtrans);
update jr;
}
}

When i tried to get code coverage on this i got below error :
Unsupported External Object Operations

LikeLike

Reply
- agarciaodeian | December 18, 2018 at 6:31 am
  
  I didn’t have the time to give it a try yet (promise I will do) but in the meantime I found this: https://developer.salesforce.com/forums/?id=9060G0000005NhiQAE
  
  LikeLike
  
  Reply
Sudarshan | November 30, 2018 at 9:05 am

How we can create Big object programmatically using Metadata API. Does Metadata API can be called from lightning/apex?

LikeLiked by 1 person

Reply
- agarciaodeian | December 18, 2018 at 6:28 am
  
  I may be wrong but as far as I know you cannot create programatically a big object. Metadata API helps you to create the object on package.xml. Please look at How to Create a BigObject? section.
  However if you are talking about creating a record, please, keep in mind the new method database.insertImmediate(). You can do it on apex, and although I didn’t test it, you can also do it on lightning components on the back end controler.
  
  LikeLike
  
  Reply