These credentials can be anything, but we need to keep note of it as we require it in next stages. Since Splunk will ask to manually type the username and password, we are using the expect package to automate the process. This fill will run commands to accept Splunk’s license and setup admin user for running Splunk commands on the container. Additionally, we are running commands in /splunk/auth.sh. In the above Dockerfile, we’ve downloaded and installed the Splunk universal forwarder package. splunkforwarder-9.0.0. \ & rm -f splunkforwarder-9.0.0. # PORTS for Splunk Universal Forwarder EXPOSE 9997 EXPOSE 8000 # Accept Splunk license and setup admin user RUN deploy/splunk/auth.sh ENTRYPOINT # Dockerfile # Splunk Universal Forwarder RUN wget \ & apt-get install -f. We can use docker entrypoint to run Splunk commands when starting the container. We’ll have to configure forward server, add monitors and start Splunk forwarder when we run the docker container. We can download & install the Splunk universal forwarder package, expose the required ports, accept Splunk license and setup the Splunk admin user when creating a docker image. I manually download the Splunk package and run Splunk commands when building the Docker image and when running the container. However, for simplicity, I opted not to use the docker image provided by Splunk. We could potentially run this as a sidecar container alongside the app container or probably use docker-compose and add it as a dependency to our application service. The default index is mainįor dockerized applications, Splunk provides a docker image for universal forwarder. my-app/log/production.log -index my-app-prod-indexHere, my-app-prod-index is an index which is manually created in Splunk. Monitor log files on certain index sudo /opt/splunkforwarder/bin/splunk add monitor. Use username and password setup in step 4Īdd your Splunk Cloud forward server sudo /opt/splunkforwarder/bin/splunk add forward-server :9997 scp /path/to/file/splunkclouduf.spl you have pem file, use scp -i file_to_copy_local_path Splunk credentials using sudo /opt/splunkforwarder/bin/splunk install app. Restart Splunk sudo /opt/splunkforwarder/bin/splunk restartĭownload the Splunk forwarder credentials file from the Splunk universal forwarder documentation link Ĭopy Splunk credentials file (downloaded in step6) from your local machine to the EC2 server using Secure Copy Protocol scp, Use following command on your local machine. You’ll be asked for a local username and password, Keep note of it, as we’ll need it later. Start Splunk by accepting license sudo /opt/splunkforwarder/bin/splunk start -acept-license. Install the package using sudo dpkg -i b This typically install the splunkforwarder on /opt/splunkforwarder for Debian based ubuntu wget -O splunkforwarder-9.0. "" You could use following steps to configure the Splunk Universal Forwarder on a EC2 machine or any other VPS servers.ĭownload the splunkforwarder package for EC2 OS from Splunk Downloads web page e.g. Configuring Splunk Universal Forwarder on EC2 Unlike HTTP Event Collector, Universal Forwarder are lightweight agents which are installed as a package on the host machine which periodically monitors the log file in the background and pushes them to the Splunk Cloud. Since HTTP Event Collector relies on HTTP connections, it might impact the application performance. While Splunk provides something called HTTP Event Collector allowing us to push data from servers to Splunk cloud over HTTP/HTTPs, I opted to use the universal forwarder. Splunk provides different type of forwarders, universal forwarder, heavy forwarder and light forwarder. To push logs to Splunk cloud, we’d need to use one of Splunk forwarders. ![]() Splunk forwarders send data from data sources to Splunk cloud for indexing which makes it easier for searching, querying and building dashboards. Different Ways of Forwarding Logs To Splunk This article is simply a documentation of the steps I took to configure Splunk universal forwarder to forward application logs from a EC2 server and a dockerized app deployed on ECS Fargate. ![]() I wanted to try and understand what goes into configuring the Splunk universal forwarder. Using Splunk queries, we could query those log data to debug any production issues as well as to build different charts and dashboards for both engineering and business reporting.Įven though I was fairly comfortable with writing Splunk queries to build dashboards utilizing the log data, I’ve never actually configured servers to forward logs to Splunk. At my day job, we use it extensively to aggregate application logs hosted on different servers. Splunk is a platform for aggregating, indexing, searching & analyzing logs and other machine generated data. Configuring Splunk Cloud Universal Forwarder
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |