Bringing data into Splunk

-
Now, lets dive deep into bringing data into Splunk.
  1. Splunk Enterprise can index any type of data, however it works best with data with timestamps.
  2. When Splunk indexes data, it breaks it into events based on timestamps.
  3. Every event or data which is indexed into Splunk should have a sourcetype ( helps to identify the type of data which is indexed ) assigned to it.
  4. In corporate environment, majorly forwarders (ref: here) are used to input data into Splunk but there are other ways as well in which you can get your data indexed to Splunk.
Lets assume you want to monitor a log file of the local machine on which Splunk is installed then you can use the hyperlinks which are listed under "Local inputs" otherwise you can use the hyperlinks which are listed under "Forwarded inputs".

For achieving that, you can navigate to "Settings" => "Data Inputs" => "Local Inputs" => "Add New" (NOTE: Make sure Splunk have access to the log files which you are trying to monitor under Splunk).

Enter the log file which you want to monitor, also select the type of monitoring which you want to achieve i.e. you just want to index the logfile once? or you want to continuously monitor mentioned log file (Just to add, this works as tail in linux as soon as there is something in log file it will index that part). Once done click on "Next".

Amazingly :), Splunk automatically divide the data into small chunks of data called events that too without any human intervention. Clink on "Next".

Select the app context, as of now we are going with "Search and Reporting", give a meaningful host name, select the index in which you want the data to be indexed. Since I already have a index called idx_messages I will choose that otherwise you can create one as per your need as well. Click on "Review".

On the final screen, if you feel everything is fine, Click on "Submit", your data is now being indexed.

Go ahead and verify via search head if your data came in or not ;)

Another thing to add, there are multiple types of source_types available in Splunk just to give you a hint below are some of them:


Comments

Post a Comment

Popular posts from this blog

#3 Splunk sub(Commands) [eval, round, trim, stats, ceil, exact, floor, tostring]

-
Image
ROUND: Eval with round takes one or two numeric arguments, returning 'first value' rounded to the amount of decimal places specified in 'second value'.  By default it will remove all the decimals. Example: index=idx_messages sourcetype=linux_logs | eval new_rt= trim ( replace (response_time, "ms.", "")) | stats avg (new_rt) as Average | eval Average= round (Average) Remove all the decimal values. Example: | eval Average= round (Average,3) Rounded the value up-to 3 decimal places. Example: | eval Average= ceil (Average) Round the value up-to the next highest integer. Example: | eval Average= exact (Average) Give the output with maximum possible number of decimal values. Example: | eval Average= floor (Average) Round the value down to the nearest whole integer. Apart from this, there are other functions as well which are used by eval command, for instance pi (), sqrt () etc. TOSTRING: Hel
Read more

#6 Splunk sub(Commands) [fields, rename, replace, table, transaction]

-
FIELDS:  This command helps to keep or remove specified fields from the search results, below command will keep just three fields in your search result. Example: | fields request, rc, pt RENAME:  This command helps to rename field(s), below command will rename a field named as service to serviceType and RC as responseCode Example: | rename service AS serviceType, RC AS responseCode REPLACE:  This command helps to replace the values of fields with another value, below command will replace the values "fetchReport" and "viewReport" as "Report" in "serviceType' field. Example: | replace fetchReport with Report, viewReport with Report in serviceType TABLE: This command helps to format the results into tabular output. Example: | table request, rc, pt TRANSACTION:  This command helps to merge events into a single event based upon a common identifier, below command will create events based on two events i.e. it will fetch the txn-id w
Read more

#1 Splunk sub(Commands) [top, rare, fields, table, rename, sort]

-
Image
TOP: Will show you top results with respect to your field. Example: index=_internal | top limit=5 component RARE: Will help you to find out least common values of a field, i.e. it is similar to TOP but works in opposite direction. Example: index=_internal | rare limit=5 component FIELDS: Will help you to limit your columns, lets say you want to remove count from above table, fields can help you to achieve that. Though there are other usage of fields as well but you will learn slowly and gradually when you start building some complex queries. Example: index=_internal | top limit=5 component | fields component, percent TABLE: Same thing can be achieved via table as well. Example: index=_internal | top limit=5 component | table component, percent RENAME: Lets say you want to rename a column, for that you can use rename command. Example: index=_internal | top limit=5 component | rename percent AS percentage | t
Read more