How to Use Elasticsearch for Search and Analytics on Kamatera

Elasticsearch is a powerful, distributed search and analytics engine designed for fast and scalable data retrieval, fine-tuned relevancy, and efficient data analysis. It is widely used for full-text search, log and event data analysis, security intelligence, and business analytics. 

As the core of the free ELK (Elasticsearch, Logstash, Kibana) stack, it enables organizations to store, search, and analyze large volumes of structured and unstructured data in real time. Its distributed nature ensures high availability, fault tolerance, and scalability, making it a preferred choice for handling massive datasets. 

This guide will walk you through installing, configuring and managing Elasticsearch on Kamatera. 

First, create an account on Kamatera. Go to console.kamatera.com and sign up for an account by providing the following information:

1. Verify your email: Click the verification link sent to your email by Kamatera.
2. Password: Enter your password.
3. Repeat Password: Re-enter your password and click Create Account.

1. Enter your credentials to access the Kamatera management console and Click Login.

2. Navigate to My Cloud on left-hand side and select Servers.

On the left-hand side navigation menu, click on Create New Server or use the Create New Server option on the right-hand side.

3. Choose zone

Choose the zone from the following options:

• Asia
• North America
• Europe 
• Middle East

 Depending on the zone you select, the available countries will be displayed. 

Use case: For this setup, we used the Asia server domain to set up the Windows Server.

4. Choose an image

Kamatera offers a variety of app and server images to help users set up preconfigured resources. Users can explore options such as 

• Server OS images
• Desktop OS images
• App images
• Service images 
• My Private images

Choose Server OS images and select Windows Server.

Note: For installing and configuring Elasticsearch, we will use the Windows Server OS image.

5. In Choose Version, select the latest version of Microsoft Windows Server (2022_standard 64-bit).

Upon selecting the version, the license prices are displayed.

6. Toggle the Detailed view button enable to view the detailed description, including the price.

For this use case, below server specifications are selected. You can configure it according to your requirements.

7. Choose Networking

Users can select the network they wish to use, whether it’s a public Internet network or a private local network.

Simple Mode

8. Advanced Configuration 

 Hide: To hide the advanced configuration. 

 Show: To see the advanced configuration.

9. Finalize Settings

Finalize settings by setting the password, re-validating it, selecting the number of servers, specifying the server name, and enabling the Power On Servers option.

10. Billing Cycle and Pricing

The user can choose between monthly and hourly billing cycles.

Note: The Server Summary displays the location, operating system (including server specifications), add-on services, servers, and pricing.

Click Create Server.

11. Click on Tasks Queue to see the progress of installation of Windows Server.

12. The server will be created and will appear on the Server Management screen. Once the server is created, click Open.

13. A new screen shows up. Overview tab in the center is selected by default and it displays information like Power state, Guest OS, Zone, Public Internet (WAN), Server ID of the server and Configuration-Number of CPU. Memory Size, Disk 1.

Click on Connect to connect to Windows Server.

14. Connection credentials like Connection Type, Username, and Password are shown. Now, click on Open Remote Console.

15. A new tab opens, connecting to the new server and displaying the Server Manager Dashboard. 

Downloading and installing Elasticsearch

1. To get started with Elasticsearch on Windows, you first need to download the latest version of Elasticsearch. Open your web browser and go to:
   https://www.elastic.co/downloads/elasticsearch 

Select Windows from the drop-down menu as shown in the screen below.

Click on the Windows zip file to download it to your system.

2. After downloading the ZIP file, by default, the file will be in your system’s Downloads folder.

Example filename: elasticsearch-8.17.3-windows-x86_64.zip

Double-click on the folder.

3. Click on Extract All as shown in the screen below or right-click on the ZIP file and select Extract All.

4. Choose a destination folder by clicking on Browse, or extract it to the default location. Click on Extract.

5. You will see that the files have been copied.

6. Now, you will see extracted folder in Downloads. Select the elasticsearch-8.17.3 -windows-x86_64 folder, right-click, and choose Copy.

Navigate to the C: drive and Paste the copied folder.

Ensure the folder is placed directly in C: for easy access.

7. You will see that the elasticsearch-8.17.3-windows-x86_64 folder has been pasted into the C: drive.

Downloading and installing Kibana

Kibana is a data visualization and exploration tool used with Elasticsearch. It allows users to analyze and visualize log data, create dashboards, and monitor real-time data trends. It is widely used for log analysis, security monitoring, and business intelligence.

1. You can download Kibana from the official Elastic website:  https://www.elastic.co/downloads/kibana. From the drop-down menu select Windows and click on the Windows button to start downloading.

2. Simply double-click on the Kibana ZIP folder to open it.

3. Inside the ZIP folder, you’ll see a directory named kibana-8.17.3 containing all necessary files. Click on Extract All as shown in the screen below or Right-click anywhere inside the folder and select Extract All.

4. Choose a destination (e.g., C:\kibana\) by clicking on Browse and click Extract. Wait for the extraction to complete.

5. Locate the extracted kibana-8.17.3 folder, right-click to copy it, and navigate to ‘C:’, where the Elasticsearch folder is already present. Paste the Kibana folder in ‘C:’.

6. Navigate to C:\elasticsearch-8.17.3\bin and double-click on the elasticsearch.bat file to start Elasticsearch. 

7. When the Security Warning window appears, click Run to proceed with executing the elasticsearch.bat file.

8. A command prompt window will open, and Elasticsearch will begin initializing. As the process runs, scroll through the output to find important security credentials. These include the Password for the elastic user, which is required for authentication, the HTTP CA certificate, used for secure communication, and the Enrollment token, which must be copied and pasted when setting up Kibana in the browser. Keep this information safe, as it is essential for accessing and configuring Elasticsearch and Kibana.

9. Navigate to C:\kibana-8.17.3\bin and locate the kibana.bat file. Double-click on it to start Kibana.

10. When the Security Warning window appears, click Run to proceed with executing the kibana.bat file.

11. A command prompt window will open, and Kibana will begin initializing. This process may take a few moments as Kibana connects to Elasticsearch and loads necessary configurations. 

The kibana.bat file will generate a unique link to access Kibana in your browser. In our case, the link was “http://localhost:5601/?code=846315”. While you can open Kibana using your specific link in any web browser, it will not be fully functional until it is registered with Elasticsearch.

12. Open your preferred web browser and enter the unique link generated during the initial setup. Upon loading, Kibana will prompt you to enter an enrollment token to proceed with the configuration. 

Paste the enrollment token generated when running the elasticsearch.bat file (refer to Step 8 screenshot). 

Click on Configure Elastic.

 When you click on “Where do I find this?” in the screenshot above, a message will appear displaying a command. Running this command will generate a new enrollment token. Once you have the token, enter it and click on “Configure Elastic” to proceed.

13. Kibana will automatically configure the necessary settings and establish a secure TLS connection with your Elasticsearch instance.

14. After the configuration is complete, you will be prompted to enter the Elastic user password, which was generated during the initial Elasticsearch setup.

Now, you are ready to use Kibana.

Configuring kibana

1. To modify Kibana’s general configuration, update the YAML-based kibana.yml file.

To do this, navigate to C:->Kibana-8.17.3->Config->Kibana.yml. This file is used by the Kibana server to load settings during startup. Right-click on the Kibana.yml file and select Open With and choose Notepad.

By default, Kibana is configured to accept local connections. It is set to use the host “localhost” and the server port 5601 (http://localhost:5601). 

2. To modify this configuration and enable remote access, locate the relevant directives in the kibana.yml file. Uncomment them by removing the # symbol and adjust the settings as needed. 

For example: server.port: 5601 and server.host: “192.168.0.102”.

3. Another important configuration is ensuring that the Elasticsearch host URL is correctly set. If Elasticsearch is not running on the same machine as Kibana (localhost), update this setting in kibana.yml to the correct domain or IP address, followed by the port number (e.g., http://elasticsearch:9200).

4. For detailed information on Kibana’s configuration directives, refer to the official Kibana documentation- https://www.elastic.co/guide/en/kibana/current/index.html 

Configuring Elasticsearch

Here, we will see how to start, stop, restart the service from Services window, set the memory limits in Elasticsearch, change the ports using the config file in elasticsearch-8.17.3 folder.

1. Now, double-click on the elasticsearch-8.17.3-windows-x86_64 folder. A new window opens, again double-click on elasticsearch-8.17.3.

Double-click on bin folder.

2. You can now run the Elasticsearch database on your computer.

If you want, you can run Elasticsearch from the command line. Let’s try doing that:

Navigate to the bin folder: C:\ elasticsearch-8.17.3-windows-x86_64\ elasticsearch-8.17.3\bin

Open Command Prompt in the bin folder by clicking on the address bar at the top and type cmd and press Enter.

3. Since elasticsearch.bat is already running, here’s how you can execute it from the command prompt. 

Open Command Prompt in the C:\ elasticsearch-8.17.3-windows-x86_64\elasticsearch-8.17.3\bin directory, allowing you to run Elasticsearch commands.

Run the below command:

Command: elasticsearch.bat

4. It may take a few moments for Elasticsearch to initialize. Once you see logs indicating that Elasticsearch is running, it is ready for use.

But the disadvantage here is that the Command Prompt window must remain open while Elasticsearch is running. If you close the terminal, Elasticsearch shuts down immediately.

5. The following command Running Elasticsearch as a Windows service ensures automatic startup whenever the system boots, eliminating the need to manually launch it. Unlike running elasticsearch.bat, the service operates in the background, so you don’t need to keep a terminal open. 

Command: elasticsearch-service.bat install

Additionally, managing Elasticsearch becomes easier with Windows Service Manager, allowing you to start, stop, or restart it easily.

6. You can see the message as “The service ‘elasticsearch-service-x64’ has been installed”.

This message indicates that Elasticsearch has been successfully registered as a Windows service. The service is now set up to run in the background, and you can manage it using Windows Service Manager or command-line tools.

7. To open the Windows Services Manager, press Windows + R on your keyboard to launch the Run dialog box. In the text field, type services.msc and click OK.

8. This will open the Services window, where you can view and manage all running services on your system. Scroll through the list to find Elasticsearch, check its status. Since the service has not been started yet, it will not show as “Running”. You will need to manually start it to activate Elasticsearch.

9. To start the service, run the below command.

Command: elasticsearch-service.bat start

You can see the message as “The service ‘elasticsearch-service-x64 has been started”.

10. Again, check whether the service has started by pressing Windows + R on your keyboard to open the Run dialog box. In the text field, type services.msc and click OK.

11. This will open the Services window, where you can view and manage all running services on your system. Scroll through the list to find Elasticsearch, check its status, and ensure it is running. From this window, you can also start, stop, or restart the service by right-clicking on it and selecting the appropriate option.

12. The following command is used to reset the password for the “elastic” built-in superuser account in Elasticsearch.

Command: elasticsearch-reset-password.bat -i -u elastic

• elasticsearch-reset-password.bat: This is a built-in script in Elasticsearch that allows you to reset passwords for internal users.
• -i: This flag enables interactive mode, where you will be prompted to confirm or set a new password.
• -u elastic: Specifies that the password reset applies to the elastic user, which is the default superuser in Elasticsearch.

You will be prompted to enter the password type ‘y’ as shown in the screen below.

Enter the password and re-enter the password. 

Then you will get a message “Password for the [elastic] user successfully reset.”

You can use this command: 

1. If you forgot the password for the “elastic” user.
2. After initial installation, if you want to manually set or change the password.
3. When configuring security settings in Elasticsearch.

13. To access Elasticsearch, open your browser and enter https://localhost:9200 in the address bar. Since Elasticsearch runs on port 9200 by default, this URL allows you to connect to the local instance.

Your browser may display a security warning, click on Advanced.

14. Click on Proceed to localhost(unsafe).

15. You will be prompted to enter your username and password to access Elasticsearch. Enter the Username as elastic along with the password you recently reset. Once you enter your credentials, click on “Sign In” to authenticate.

16. If the login is successful, you will see a JSON response displaying key details about your Elasticsearch instance, such as its version, cluster name, etc.

17. Check the service by pressing Windows + R on your keyboard to open the Run dialog box. In the text field, type services.msc and click OK.

18. This will open the Services window, where you can view and manage all running services on your system. Scroll through the list to find Elasticsearch, check its status, and ensure it is running. Now, by default, Elasticsearch runs as a Windows background service. 

But if the startup type is set to Manual, you will need to start the service manually each time your PC restarts. 

To change this setting, double-click on Elasticsearch, locate the Startup type field, and select Automatic or Automatic (Delayed Start) from the drop-down menu. This ensures that the Elasticsearch service starts automatically whenever the system reboots. 

19. Change the folder where data is stored:

By default, Elasticsearch stores its data in the data folder within the Elasticsearch installation directory as shown in the screen below.

To modify the existing data folder or add additional storage locations, update the elasticsearch.yml configuration file. Let’s see how to do this.

20. Go to Local Disk (C)-> elasticsearch-8.17.3->config.

Double-click on config folder.

21. Right-click on the elasticsearch.yml file and select Open With to choose Visual Studio Code or Notepad.

Note: In this example, the elasticsearch.yml file is opened using Visual Studio Code.

22. Locate the path.data setting in the elasticsearch.yml file and configure it to use a single or multiple data directories.

23. Remove the # symbol before path.data to uncomment the setting.

To set a single data folder, use: path.data: “C:\elasticsearch-8.10.2\data”.

After making these changes, save the file and restart the Elasticsearch service for the updates to take effect.

24. Now, restart the service in Windows Services Manager, by pressing Windows + R on your keyboard to launch the Run dialog box. In the text field, type services.msc and click OK.

25. This will open the Services window, where you can view and manage all running services on your system. Scroll through the list to find Elasticsearch and restart the service by right-clicking on it and selecting the appropriate option.

26. To specify multiple data folders:

path.data: [“C:\elasticsearch-8.10.2\data”, “D:\elastic_data”]

After making these changes, save the file and restart the Elasticsearch service for the updates to take effect.

27. Set the memory limit of Elasticsearch windows service:

Elasticsearch Windows service defaults to 1 GB of memory, which may cause errors with large queries. To increase the limit, open Command Prompt in the Elasticsearch folder and run:

Command: elasticsearch-service.bat manager

This opens a new window, allowing you to adjust memory settings.

28.Go to Java tab and set the initial and maximum memory pool values to higher values, say 10 GB (=10240 MB). 

Click Ok.

29. Change the Elasticsearch port:

Go to Local Disk (C)-> elasticsearch-8.17.3->config->elasticsearch.yaml

Right-click on elasticsearch.yaml and Open With Visual Studio Code.

30. By default, Elasticsearch runs on port 9200.

31. Remove the # symbol before path.data to uncomment the setting. 

Change the port to your desired port number.

After making these changes, save the file and restart the Elasticsearch service for the updates to take effect.

32. Now, verify the new port by opening a browser and entering https://localhost:9201 in the address bar.

Kibana dashboard

1. A Kibana dashboard is an interactive interface for visualizing, analyzing, and monitoring data in Elasticsearch. It offers customizable charts, graphs, tables, and maps, enabling real-time insights, log monitoring, security analysis, and business intelligence. With built-in filtering and drill-down capabilities, Kibana simplifies data exploration for informed decision-making. 

If you want to add integrations click on Add Integrations 

otherwise click on Explore on my own to explore Kibana dashboard on your own.

2. “Add Integrations” connects Kibana to various data sources and services. On this page, “Browse Integrations” is selected by default.

3. When you click on Installed Integrations, you will see currently installed integrations.

4. Click on the three-line menu in the top-left corner of the page to access Analytics, Observability, Security, and Management.

• Analytics – Create dashboards and visualize data stored in Elasticsearch.
• Observability – Monitor system performance, logs, and application metrics.
• Security – Manage access control, threat detection, and security insights.

Management – Configure settings, data indices, and system integrations.

5. Click on “Add Integrations” as shown in the screenshot below. Add Integrations will connect Kibana with various data sources and services.

6. On the right-hand side, click on the help icon as shown in the screenshot below. This will provide access to documentation and various other options.

7. On the right-hand side, click on the “What’s New at Elastic” icon as shown in the screenshot below. It displays the latest updates and information from Elastic.

8. Now, click on the “Explore on my own” button to navigate through the features and functionalities of Kibana independently.

9. Click on Try sample data option to load pre-configured datasets into Kibana.

10. By using sample data, you can practice creating visualizations, dashboards, and performing searches within Elasticsearch, helping you get familiar with the platform before integrating real-world data. 

Click on Start exploring.

11. This opens the “Welcome to Elastic Demo Environment” page, a live, read-only environment that allows you to explore the capabilities of the Elastic platform for search, observability, and security use cases. Click on any of the icons below to get started. 

Click on icon in the Search option.

12. A new page opens, allowing you to add data to Elasticsearch and then search, vectorize, or visualize it.

13. To upload a file, go to the Home page and click on Upload a file.

14. Here, you can select a file or drag and drop it into the designated area.

And that’s it! You have successfully learned how to use Elasticsearch for search and analytics on Kamatera.