Apache Kafka is a distributed streaming platform. It is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault tolerant, wicked fast, and runs in production in thousands of companies.
Kafka fundamental
Code-server, the VSCode for cloud
Along with the widely cloud adoption, integrated development environment (IDE) on browser is a need to boot developers’ productivity. People can collaboratively view, edit and commit on any devices with internet accessed browser. Additionally, you’re no longer worry about setting up your local development config. You can consider Cloud9 (AWS) or paid service like codeanywhere.
Kibana, the elastic stack's central management
Kibana is part of ELK stack to visualize data from elasticsearch. Further than that, Kibana is equipped with many features and plug-ins such as elastic nodes & infrastructure monitoring, user roles or life cycle management and query experiment elasticsearch database.
Spend sometime with the demo Kibana page to feel it. Click Here.
Grafana, the best visualization tool for monitoring?
Grafana is an open-source web application specialized in time-series visualization. Hence, it is suitable for the the purpose of monitoring. One of interesting facts is that Grafana is a fork from Kibana 3 to enhance the ability of dashboard editing and make it as a clean and elegant time-series visualization tool. To get the sense of Grafana dashboard, surf this link: https://play.grafana.org.
Quick tour with Elasticsearch 6.x
When do researching to choose a good data storage technique for log collection, searching and analytic; I found elasticsearch is a ideal choice because of following reasons:
- Performance: fast query with million records within miliseconds, it is thanks to indexing document technique with Lucene engine running under-the-hood.
- Scalability: elasticsearch can be expanded by simply configuring new nodes when resource increase needed.
- Integration : it is compatible with elastic stacks (beats: metric, file, heart, etc. ) and others (Fluentd, grafana, etc.) which support many purposes to monitor multiple system and services.
Setup LDAP for Apache Nifi
Default Apache Nifi installation comes without security layer which exposes the development UI. As a result, users can freely access the Nifi project development with knowledge about the hostname and binding Port. You can see two potential security risks:
- Flow controller attack : full policies to modify the processor on Flow Controller.
- API attack: external invoked requests to start/stop/delete Nifi components.
Vietnam stock data analysis
The goal of this project is to collect and visualize the stock price of all tickers in Vietnam. There is quite limited access to API for a single business user, this project aim at scrap data from website, clean, extract and load into data warehouse. The final product is a maintainable/reliable data pipeline with exposed analytic dashboard hosted on cloud, and end authorized users can access to 24/7 with daily updated data.
Car make & model recognition
This project aims at recognizing the car make and model based on a Stanford Cars Dataset with 16,185 images. This dataset includes information about car make, model, and year (Eg. 2012 Tesla Model S) with 196 different classes. However, in this project we target to identify the car make and model only; this results in 164 different classes in total.
Vietnamese sign language reader
This project describes a computer based system that utilizes a 3D sensor to bridge the communication barrier between hearing (and/or speech) impaired people and hearing ones. Above figure shows how I love Vietnam demonstrated in Vietnamese Sign Language.
Entrance checking system
This project aims at implementing an intelligent human resources management system which combines the RFID and face recognition method. The proposed system will have a camera to capture the faces of people and a RFID reader to check the ID numbers. If both of the verifying processes return a “Pass” signal, then a successful entrance signal is generated.