¶ Welcome to the Splunk Lab
¶ Overview
In this lab we will teach you how to setup your own Splunk server. You will also learn how you can forward logs from different endpoint devices to your Splunk logs, aka ingesting logs. After you get comfortable ingesting logs we will also be generating our own log sources and learning how we can ingest those logs too.
¶ Technologies you will use
- Splunk Enterprise
- Splunk Universal Forwarder
- Linux - Debian
- Windows
- Sysmon
As more and more people complete the beginner lab we will start working on a more advanced lab for people interested in learning more about Splunk.
¶ Lab Structure
- We give you a task. We want you to figure out as much as you can by yourself so the instructions will be very minimal.
- If you're having trouble with a task, don't hesitate to check the hints section.
- Don't be discouraged as we also share the answers.
¶ Setup
SDC Kamino Instructions
-
First you will need to download and install Pritunl
-
Next download this vpn profile sdc_vpn.ovpn
-
Import the profile using Pritunl. Dont enter a profile url.
-
Once Imported, connect to the sdc_vpn and use the credentials below
user:vpn
password:141252
-
Open up a web browser and navigate to https://kamino.calpolyswift.org/#/
-
Register an account and login
-
Click the drop down under
Deploy Pod From Templateand selectSOC - Splunk Lab- ignore any errors about clone failed. -
Click the refresh icon when complete. Click
OPEN POD ON ELSA.SDC.CPP -
Press "Advanced" and "proceed to unsafe"
-
Login using the account you created in step 6
-
Click on the burger menu on the top left and select
Inventory -
Select the
second iconthat has 3 squares overlapping on paper -
Drill down into the menu until you see 4 VMs. Power on the VMs provided by pressing the play button or going to
Actions->Power->Power ON
You are now ready to start the lab. If you get stuck on any stage feel free to reach out Joe | Shobra on Discord.
¶ Task 1 - Setting up Splunk Server
After you complete the setup of the lab, we will now work with the vm called "Splunk Server".
Task 1 is to create your very own Splunk server. When I say create I simply mean install the Splunk Enterprise Trial(https://www.splunk.com/en_us/download/splunk-enterprise.html), run it and acces it.
Note the username is
socand password is1234
IMPORTANT: Before you start run
su -to login as the root user.
Here are some guiding qustions you should be asking yourself to get you started:
Where can I download Splunk Enterprise Trial?
How do you install a .deb package?
Where is the default location Splunk gets installed on Linux?
How do I run Splunk?
How do I access the Splunk web interface?
If you are stuck check out the hints. Good luck!
¶ Hints
Hint 1
Navigate to https://www.splunk.com/en_us/download/splunk-enterprise.html in your browser of choice.For the Splunk Server, we would want to download the .deb file. Copy the wget command in the terminal and run it to download Splunk Enterprise Trial or just click the download button.
Hint 2
In debian we use the dpkg -i packages to install packages.
Hint 3
The Splunk installer installs Splunk in /opt/splunk by default.To run Splunk you will need to find the binary called splunk and run it using the command `./splunk start`.
¶ Answer
Answer
You sure?
Normally you would login to the Splunk Portal and download the .deb file But to save on time you can use the command below to download the Splunk Enterprise Trial. Note this is the latest version as of 08/28/24 so if the command does not work you will need to update the download links by visting Splunk's download website.
wget -O splunk-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb "https://download.splunk.com/products/splunk/releases/9.3.0/linux/splunk-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb"
Navigate to the folder you downloaded the file and run the command
dpkg -i splunk-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb
Accept the license and create an account. Create a username and password. Navigate to the folder /opt/splunk/bin and run
cd /opt/splunk/bin
./splunk start
Open the link shown on your terminal e.g. http://splunkserver:8000
Login with the user account you created when installing Splunk.
On the Splunk Server navigate to the app called "Search & Reporting". Once it loads, skip the tour and you should see a huge search bar on screen. This is the main way we will be searching for the data the comes into your Splunk Server. Try searching index="main" and press enter. Notice that you do not see any data and that is expected because we have not gathered any data yet. We will be doing this in the next task.
Congratulations you have successfully installed and accessed your first Splunk Server!
If you havent already, I encourage you to take explore the UI of Splunk. It will be quite a lot to absorb at first but I promise you as you keep using Splunk you will get used to the UI.
¶ Task 2 - Forwarding Logs to Splunk
Congratulations if you made it this far that means you successfully setup your own Splunk server!
Task 2 is to install the Splunk Universal Forwarder on the Windows Endpoint and see if you can query the logs using the Splunk Search App. Go back to the tab you used to access the Splunk Server and power on and connect to the Windows Endpoint.
Now that we set up a Splunk server we need to figure out how we can ingest logs into it. We ingest logs using something called the Splunk Universal Forwarder(https://www.splunk.com/en_us/download/universal-forwarder.html). This is a tool installed on an endpoint device that tells the computer which logs to send and where to send them.
If you are able to see data when searching index=main on the Splunk Search App you have successfully completed the task.
Here are some guiding qustions you should be asking yourself to get you started:
Where can I download Splunk Universal Forwarder?
How do you install Splunk Universal Forwarder on Windows?
How do I check if the Splunk Universal Forwarder is running?
What firewall ports number does Splunk Universal Forwarder use?
What communication protocol does Splunk Universal Forwarder use?
How do I open the firewall port on Windows?
How do I receive data on the Splunk Server?
IMPORTANT: When installing the Splunk Universal Forwarder click on the Customize Options and during one of the steps make sure you select the Local System option. ALSO SKIP THE DEPLOYMENT SERVER IP ADDR. This is something different.
¶ Hints
Hint 1
You can download the Splunk Universal Forwarder at https://www.splunk.com/en_us/download/universal-forwarder.html.- Run the msi file
- Accept the license and click `Customize Options`
- Click next to to skip directory
- Click next to skip SSL
- Select Local System and click next
- Select Application, Security, System logs and click next
- Enter any username and press next
- Click next to skip Deployment Server
- Enter the IP address of your Splunk Server (which is not 127.0.0.1) and port 9997 and click next.
- After it is installed click Finish
The Splunk Universal Forwarder should now be running in the background to check if it is running you can click the Windows icon search Services, open the Services app. Scroll down to find SplunkForwarder, it should be in the Running state.
If you made a mistake you can always uninstall the Splunk Universal Forwarder by going to Control Panel -> uninstall a Program and uninstall Splunk Universal Forwarder.
Hint 2
Did you configure your Splunk server to receive data? Perhaps there is a setting in the Splunk Server that allows us to setup a receiver on port 9997.Did you open up firewall ports? Splunk Universal Forwarders use TCP port 9997 to communicate with the Splunk Server. The Splunk Server has no firewall installed so you do not have to deal with ports. Unless it's a Windows server, in which case you do need to make a firewall rule.
Hint 3
If you are the Windows endpoint and you want to communicate out to the Splunk Server. Would you need to create an inbound rule or outbound rule?If you think you did everything correctly you should be able to query index="main" in your Splunk Server to see the data.
¶ Answer
Answer
You sure?
Download the Splunk Universal Forwarder. You can either log into the Splunk website with your account and download the windows version. Or do the same with this command
wget -O C:\splunkforwarder-9.3.0-51ccf43db5bd-x64-release.msi "https://download.splunk.com/products/universalforwarder/releases/9.3.0/windows/splunkforwarder-9.3.0-51ccf43db5bd-x64-release.msi"
This will downloaded the installer to you C:\ folder.
- Run the msi file
- Accept the license and click `Customize Options`
- Click next to to skip directory
- Click next to skip SSL
- Select Application, Security, System logs and click next
- Enter any username and press next
- Click next to skip Deployment Server
- Enter the IP address of your Splunk Server (which is not 127.0.0.1) and port 9997 and click next
- After it is installed click Finish
Open up Windows Firewall and open the Advanced options. Create a new outbound rule for port TCP 9997 and allow the connection. Do the same but for an inbound connection if you are using a Windows server to run Splunk Enterprise.
On your Splunk Server login using the credentials you created and go to Settings->Forwarding and Receiving and under the Receive Data section click on +Add New. Enter 9997 as listening port and save.
Go to your Splunk Search Head and search index=main.
You should see data in this index after a few minutes.
¶ Task 3 - Forwarding Linux Logs to Splunk
Nicely done comrade! You have now successfully forwarded the logs from a Windows device. The Windows logs you have forwarded to Splunk can be found locally in C:\WINDOWS\system32\winevt\Logs. I encourage you to navigate there yourself to see what else is available. Also you can use Event Viewer on Windows to make it easier to view logs.
In Task 3 you will have to do the same thing you did in Task 2 but now for Linux!
Keep in mind that the Linux logs are named differently and stored in a different folder than Windows.
Here are some qustions you should be asking yourself to get you started:
- How do I install the Splunk Universal Forwarder on Linux?
- How do I check that the Splunk Universal Forwarder is running on Linux?
- What command do I use to run the Splunk Universal Fowarder?
- How do I tell the Splunk Universal Forwarder to send logs to a Splunk Server?
- How do I tell Splunk Universal Fowarder to monitor a folder?
¶ Hints
Hint 1
You can download the Splunk Universal Forwarder at https://www.splunk.com/en_us/download/universal-forwarder.html.Similar to the Splunk Server install you use the dpkg -i to install the Splunk Universal Forwader on Debian.
Start the Splunk Universal Forwarder and check that it is running using the command systemctl status SplunkForwarder.service
Hint 2
We now need to run commands to do the following. We first need to tell the Splunk Universal Forwarder where to send our data. Use command `/opt/splunkforwarder/bin/splunk add forward-server IPADDR` Next we will need to find the location where linux logs are stored. Use command `/opt/splunkforwarder/bin/splunk add monitor FOLDERNAME`Hint 3
If it still is not working try restarting the splunkforwarder using `systemctl restart splunkforwarder.service`¶ Answer
Answer
You sure?
Download the Splunk Universal Forwarder. You can either log into the Splunk website with your account and download the windows version. Or do the same with this command
wget -O splunkforwarder-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.3.0/linux/splunkforwarder-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb"
- Run `su -` to change into root
- Navigate to the folder you downloaded the file
- Run `dpkg -i splunkforwarder-9.3.0-51ccf43db5bd-linux-2.6-amd64.deb`
- Run `/opt/splunkforwarder/bin/splunk start`
- Run `/opt/splunkforwarder/bin/splunk add forward-server IPADDR`. The IPADDR is the ip address of your Splunk Server
- Run `/opt/splunkforwarder/bin/splunk add monitor /var/log/`
Go to your search app in the Splunk Server and run index=main. If you see another host with the name deb12 you have successfully completed this task.
¶ Task 4 - Custom Indexes
Wow you really must like Splunk if you came this far 
Task 4 will be a quick and easy one.
Notice how all your indexes are being sent to main by default. Imagine ingesting 100s of log sources into main. How messy does that look. It would also be a nightmare querying specific data and very inefficient.
Task 5 is to create new indexes and change the behaviour of the Universal Forwarder.
Create an index called Windows and another one called Linux. Our goal is to send the logs from Windows to the Windows index and the logs from Linux to the Linux index.
Here are some qustions you should be asking yourself to get you started:
- How do I create an index in Splunk?
- Which file in does Splunkforwarder use to know which logs to send?
- How do I tell Splunkforwarder to send logs to another index?
- How do I restart the Splunkforwarder?
¶ Hints
Hint 1
There should be a setting that allows us to create new indexes in the Splunk Server Gui.Hint 2
The file you want to be editing is the inputs.conf file? Be careful there may be more than one inputs.conf file but keep in mind that Splunk reads config files based on this order- 1. System local directory -- highest priority
- 2. App local directories
- 3. App default directories
- 4. System default directory -- lowest priority
Hint 3
If you made successfully made changes to the config file dont forget to restart your Splunkforwarder!¶ Answer
Answer
You sure?
Create a new index by going to Settings -> Indexes and clicking the button on the top right called "New Index". Enter Index Name as "Windows" and save. Create another one called "Linux". > the name of the index should be exact!In Windows you can find the inputs.conf under 'C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local'
Open notepad as Administrator and open inputs.conf
Add the line index = Windows under each section of logs. e.g.
[WinEventLog://Application]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = Windows
Save the file
Open the Services app and restart the Splunk Universal Fowarder.
In Linux you can find the inputs.conf under '/opt/SplunkUniversalForwarder\etc\apps\search\local'
Run 'nano inputs.conf' to dedit
Add the line index = Windows under each section of logs. e.g.
[monitor:///var/log]
disabled = 0
index = Linux
Save the file
Open the Services app and restart the Splunk Universal Fowarder.
¶ Task 5 - Generating Custom Log Sources
If you come this far pat yourself on the back!
You essentially now have a good understanding on how logs are ingested into Splunk and you have a basic understanding of how to change the flow of data.
Remember how easy it was when you forwarded logs using the Windows forwarder o forward data using the settings in the nice GUI? Well what if we wanted to send logs from other programs that are not listed in the installer? The good news is that since you fundamentally know how to send logs and which file controls that we can forward any data we want! Thats crazy!!
Task 5 is to install [Sysmon] on your Windows Endpoint (https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon) with SwiftOnSecurity configuration and ingest the logs Sysmon generates into Splunk.
If you are able to see WinEventLog:Microsoft-Windows-Sysmon/Operational in the Sources Field when searching index=main you have successfully completed the task.
¶ Hints
Hint 1
You can download Sysmon at https://learn.microsoft.com/en-us/sysinternals/downloads/sysmonAfter you install Sysmon you need to load the SwiftOnSecurity config file. See github instructions.
Hint 2
Try googling for "How to forward Sysmon logs to Splunk"Hint 3
Did you remember to restart the Splunk Universal Forwarder?¶ Answer
Answer
You sure?
Download Sysmon and extract the folder.Download the SwiftOnSecurity config file onto your Windows endpoint and copy it into the extracted folder with the Sysmon64.exe file. Open up powershell and navigate to where the Sysmon64.exe files is.
If you are installing with new config use
.\Sysmon64.exe -accepteula -i sysmonconfig-export.xml
If you already installed Sysmon using default config use this command to update the config.
.\Sysmon64.exe -c sysmonconfig-export.xml
Add the following lines to your inputs.conf file that you are using
[WinEventLog://Microsoft-Windows-Sysmon/Operational]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = main
Restart the Splunk universal forwarder service
If you enjoyed this lab and want to learn more feel free to reach out to us on Discord!
¶ Whats next?
Congratulations you have completed the Splunk Lab!
Hope you enjoyed the lab! If you want to work on more Splunk stuff or like what you see at the SOC, I encourage you to get involved and start working on your own cool project 
. You are now dubbed Splunk Pro!
Stay tuned for a more advanced Splunk Lab where we plan to tackle the following questions:
-How do I scale my Splunk deployment?
-How do I make my Splunk server more resilient?
-How do I increase search performance?