27 April 2018

Scaling Splunk with the Qumulo File Fabric

Splunk is a market leading platform for machine data. It allows to gather all kinds of log and machine generated data in a scalable manner to index, analyze, visualize large data sets. It provides historic and real time data analytics and a large ecosystem around it, including Machine Learning libraries and many more tools.
Slide1
Figure 1: Splunk harnesses machine data of any kind for indexing, searching, analysis etc.


Architecture

The main components of any Splunk implementation are Forwarders, Indexers and Search Heads. Forwarders are typically software agents that run on the devices to monitor and forward steams of logs to the indexers. Indexers are the heart of Splunk’s Architecture. This is where data is parsed and
indexed in real time. Search heads are separate servers to which users connect to query data, build reports and visualize data (in smaller environments indexers and search heads can run on the same servers).

Slide2
Figure 2: Splunk Architecture Components: Forwarders, Indexers, Search Heads

Data Tiering

Data in Splunk is stored in buckets:
  1. Hot Buckets: this is where data is stored at arrival. Hot buckets are kept open for writing until a certain threshold is reached. Then a hot bucket is being closed and moved to a warm bucket.
  2. Warm Buckets: Warm buckets contain are also in the index for searching and data can still be written to them. When the threshold for warm bucket capacity is reached, older warm buckets are being moved to the storage for cold buckets.
  3. Cold Buckets hold the majority of the data in most cases. Cold buckets are read only but are still in the index. Thus, cold buckets will appear in all search results, reports etc.
  4. Frozen Buckets are buckets that are not in the index anymore and are stored for archive purposes only. They are useless for searching/analysis and reporting.

Slide3
Figure 3: Splunk Buckets


Qumulo Universal-Scale NAS to improve efficiency

Splunk can use local Storage or Direct Attached Storage (DAS) for all bucket types. However, this is relatively inefficient. If reliability is required, the Replication Factor (RF) and the Search Factors (SF) need to be increased. The Replication Factor indicates how many replicas are being held for the raw data while the Search Factor determines the number of copies for the index data. Both have a default value of two but can be changed at implementation time. A factor of two means that all stored data is doubled.
In addition, DAS storage is complex to manage. Whether you are using stupid JBODs or RAID arrays, in both cases there is a significant administration overhead. Rebuild times are extremely long in traditional RAID arrays which translates to increased risk of data loss.
A much better solution for the majority of data sitting in cold buckets is Qumulo’s Universal-Scale Filesystem QF2. It is a Software Defined Storage Solution that can be deployed on x64 based servers (i.e. from Qumulo and 3rd party vendors like HPE) or in the Cloud.


Qumulo’s Hypbrid Architecture

QF2 has a unique Scale-Out Architecture that starts with four nodes and it scales to many petabytes of capacity by adding nodes. It utilizes a hybrid model where SSDs are being used to build a relatively large write and read caching layer and HDDs are being used to store colder data. Thanks to this hybrid architecture, all writes and many reads are directly being served from SSDs but the economics is largely dictated by the large HDDs that Qumulo servers use.

Slide4


Summary and Benefits

QF2 provides an almost bottomless pool of capacity that is extremely easy to manage
  • The capacity can be scaled as needed by adding additional nodes
  • Processing power and be scaled independently from storage. More users or more complex query will increase processing power but not storage.
  • Frozen Buckets can be avoided as data can be stored on efficient QF2 at an attractive price level in cold buckets. Data remains searchable. Storing more Splunk data allows you to run query against data covering many years of data rather than your data from the last couple of month. This provides a more accurate view of trends as well as anomalies.
  • Simplification: a Qumulo cluster is manages effortlessly compared to many DAS instances.
    Instead of increasing Splunk’s replication factor to increase availability, data in QF2 is protected by a much more efficient erasure coding.
  • Snapshots can be used to effectively backup data.

Links

Futher detail on Qumulo’s Universal-Scale Filesystem can be found here:
The Promise of Universal Scale (White Paper). It’s a high level, marketing oriented White Paper. https://qumulo.com/documents/21/WP-Q151-Promise-of-Universal-Scale.pdf
Qumulo File Fabric Technical Overview (with good detail on data protection): https://qumulo.com/documents/20/WP-Q152-QF2-Technical-Overview.pdf

20 comments:

  1. Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    emc software vendors
    bmc software vendors
    Microsoft goldpartner
    sap crm service providers

    ReplyDelete

  2. such a wonderful article...very interesting to read ....thanks for sharining .............
    Hadoop online training in pune

    Hadoop training in mumbai

    Bigdata Hadoop training in usa

    ReplyDelete
  3. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Anexas
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

    ReplyDelete
  4. Al-Fares International Tents knows how significant your event is to you, either it's a Ramadan Tent or Iftar Tent, we will work with you in as large or small a capacity as you need. Ramadan Tents | Party Tents | Event Tents | Wedding Tents.
    Exhibition Tents Rental Dubai | Tent Rental Dubai | Tent Manufacturer Dubai | Tent Supplier Dubai | Tent Rental Company Dubai | Event Tent Rental Dubai | Wedding Tent Rental Dubai | Outdoor Tent Rental Sharjah | Warehouse Tent Rental Abu-Dhabi | Tents and Marquees Supplier | Tents Manufacturers UAE

    ReplyDelete
  5. IMPRESSED WITH SUCH A GOOD CONTENT!!
    VERY INTERESTING
    GREAT WORK
    network solutions in dubai

    ReplyDelete
  6. IMPRESSED WITH SUCH A GOOD CONTENT!!
    VERY INTERESTING
    GREAT WORK
    nas storage dubai

    ReplyDelete
  7. IntelliMindz is the best IT Training in Bangalore with placement, offering 200 and more software courses with 100% Placement Assistance.

    Splunk Training In Bangalore
    Building Estimation and Coasting Course In Bangalore
    TestComplete Training In Bangalore

    ReplyDelete
  8. While I have read your article several times, I find many valid points in it. I am confident your readers will enjoy it. Travelers can apply for a Turkey e visa which is very easy. If you have an internet connection and valid documents, you can apply online from anywhere in the world.

    ReplyDelete
  9. The DNV Equipment in light of Delhi production and provider. This organization upvc equipment providers in india and reliably serving for that large number of individuals who in a real sense need to get the best out of the UPVC embellishments best upvc doors and windows delhi providers on the lookout. The Metalkraft Window Adornments in light of Hyderabad assembling and provider. This organization providers upvc equipment in india. Fates Equipment gives you a total scope of UPVC entryways and windows equipment. This organization in view of delhi provider.

    ReplyDelete
  10. Thanks for sharing the amazing blog with us, find the best Mobile & Tablet Security Display Stands for your any purpose.

    ReplyDelete
  11. Thank you for sharing useful information with us. Please keep sharing like this. Looking for top-quality CBSE online tuition classes? Join Ziyyara’s leading platform for CBSE tuition and unlock your full academic potential from the comfort of your home.
    For more info contact +91-9654271931 or visit CBSE Online Tuition Near Me

    ReplyDelete