Docker Timezone Settings

【Notes】Docker Timezone Settings

If we want to Dockerset hostthe same time zone internally, how should we set it?

Quick Start

Many people on the Internet said that containeryou can add the following command at startup:

  • Method 1:

    -v /etc/localtime:/etc/localtime:ro

  • Method 2:

    -and “TZ=Asia/Taipei”

But sometimes it works, and sometimes it doesn’t work. Why? It depends on what you usedocker image

Use alpinethe version

dockerfile

 dockerfile 364 B 
12345678910111213
FROM openjdk:12-alpine
VOLUME /tmp

RUN apk update && \
    apk add -U tzdata

ADD build/libs/sb-crawler-0.0.1-SNAPSHOT.jar app.jar
RUN sh -c 'touch /app.jar'

EXPOSE 6565
ENV JAVA_OPTS=""
ENV SPRING_PROFILES_ACTIVE=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/urandom -Dspring.profiles.active=$SPRING_PROFILES_ACTIVE -jar /app.jar" ]
  • We dockerfileadded the installation tzdatainstructions in
RUN apk update && \
    apk add -U tzdata

Start command

  • Add to-v /etc/localtime:/etc/localtime:ro
docker run --name sb-crawler -d -h sb-crawler -p 8081:8081 -e SPRING_PROFILES_ACTIVE=prod -v /home/ec2-user/log/sb-crawler:/log -v /etc/localtime:/etc/localtime:ro registry.gitlab.com/sportsbook1/sb-crawler:$CI_COMMIT_SHORT_SHA

Use alpinea non-

The disadvantage is that it is alpinemuch larger than the previous version and takes more time (push image)

dockerfile

  • Used imageforopenjdk:12

Start command

  • Add to-e "TZ=Asia/Taipei"

That’s it for today’s notes. If you have any questions, let me know!

A Tour of Python NLP Libraries

NLP, or Natural Language Processing, is a field within Artificial Intelligence that focuses on the interaction between human language and computers. It tries to explore and apply text data so computers can understand the text meaningfully.

As the NLP field research progresses, how we process text data in computers has evolved. Modern times, we have used Python to help explore and process data easily.

With Python becoming the go-to language for exploring text data, many libraries have been developed specifically for the NLP field. In this article, we will explore various incredible and useful NLP libraries.

So, let’s get into it.

NLTK

NLTK, or Natural Language Tool Kit, is an NLP Python library with many text-processing APIs and industrial-grade wrappers. It’s one of the biggest NLP Python libraries used by researchers, data scientists, engineers, and others. It’s a standard NLP Python library for NLP tasks.

Let’s try to explore what NLTK could do. First, we would need to install the library with the following code.

pip install -U nltk

 

Next, we would see what NLTK could do. First, NLTK can perform the tokenization process using the following code:

import nltk from nltk.tokenize
import word_tokenize

# Download the necessary resources
nltk.download('punkt')

text = "The fruit in the table is a banana"
tokens = word_tokenize(text)

print(tokens)

 

Output>> 
['The', 'fruit', 'in', 'the', 'table', 'is', 'a', 'banana']

 

Tokenization basically would divide each word in a sentence into individual data.

With NLTK, we can also perform Part-of-Speech (POS) Tags on the text sample.

from nltk.tag import pos_tag

nltk.download('averaged_perceptron_tagger')

text = "The fruit in the table is a banana"
pos_tags = pos_tag(tokens)

print(pos_tags)

 

Output>>
[('The', 'DT'), ('fruit', 'NN'), ('in', 'IN'), ('the', 'DT'), ('table', 'NN'), ('is', 'VBZ'), ('a', 'DT'), ('banana', 'NN')]

 

The output of the POS tagger with NLTK is each token and its intended POS tags. For example, the word Fruit is Noun (NN), and the word ‘a’ is Determinant (DT).

It’s also possible to perform Stemming and Lemmatization with NLTK. Stemming is reducing a word to its base form by cutting its prefixes and suffixes, while Lemmatization also transforms to the base form by considering the words’ POS and morphological analysis.

from nltk.stem import PorterStemmer, WordNetLemmatizer
nltk.download('wordnet')
nltk.download('punkt')

text = "The striped bats are hanging on their feet for best"
tokens = word_tokenize(text)

# Stemming
stemmer = PorterStemmer()
stems = [stemmer.stem(token) for token in tokens]
print("Stems:", stems)

# Lemmatization
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(token) for token in tokens]
print("Lemmas:", lemmas)

 

Output>> 
Stems: ['the', 'stripe', 'bat', 'are', 'hang', 'on', 'their', 'feet', 'for', 'best']
Lemmas: ['The', 'striped', 'bat', 'are', 'hanging', 'on', 'their', 'foot', 'for', 'best']

 

You can see that the stemming and lentmatization processes have slightly different results from the words.

That’s the simple usage of NLTK. You can still do many things with them, but the above APIs are the most commonly used.

SpaCy

SpaCy is an NLP Python library that is designed specifically for production use. It’s an advanced library, and SpaCy is known for its performance and ability to handle large amounts of text data. It’s a preferable library for industry use in many NLP cases.

To install SpaCy, you can look at their usage page. Depending on your requirements, there are many combinations to choose from.

Let’s try using SpaCy for the NLP task. First, we would try performing Named Entity Recognition (NER) with the library. NER is a process of identifying and classifying named entities in text into predefined categories, such as person, address, location, and more.

import spacy

nlp = spacy.load("en_core_web_sm")

text = "Brad is working in the U.K. Startup called AIForLife for 7 Months."
doc = nlp(text)
#Perform the NER
for ent in doc.ents:
    print(ent.text, ent.label_)

 

Output>>
Brad PERSON
the U.K. Startup ORG
7 Months DATE

 

As you can see, the SpaCy pre-trained model understands which word within the document can be classified.

Next, we can use SpaCy to perform Dependency Parsing and visualize them. Dependency Parsing is a process of understanding how each word relates to the other by forming a tree structure.

import spacy
from spacy import displacy

nlp = spacy.load("en_core_web_sm")

text = "SpaCy excels at dependency parsing."
doc = nlp(text)
for token in doc:
    print(f"{token.text}: {token.dep_}, {token.head.text}")

displacy.render(doc, jupyter=True)

 

Output>> 
Brad: nsubj, working
is: aux, working
working: ROOT, working
in: prep, working
the: det, Startup
U.K.: compound, Startup
Startup: pobj, in
called: advcl, working
AIForLife: oprd, called
for: prep, called
7: nummod, Months
Months: pobj, for
.: punct, working

 

The output should include all the words with their POS and where they are related. The code above would also provide tree visualization in your Jupyter Notebook.

Lastly, let’s try performing text similarity with SpaCy. Text similarity measures how similar or related two pieces of text are. It has many techniques and measurements, but we will try the simplest one.

import spacy

nlp = spacy.load("en_core_web_sm")

doc1 = nlp("I like pizza")
doc2 = nlp("I love hamburger")

# Calculate similarity
similarity = doc1.similarity(doc2)
print("Similarity:", similarity)

 

Output>>
Similarity: 0.6159097609586724

 

The similarity measure measures the similarity between texts by providing an output score, usually between 0 and 1. The closer the score is to 1, the more similar both texts are.

There are still many things you can do with SpaCy. Explore the documentation to find something useful for your work.

TextBlob

TextBlob is an NLP Python library for processing textual data built on top of NLTK. It simplifies many of NLTK’s usage and can streamline text processing tasks.

You can install TextBlob using the following code:

pip install -U textblob
python -m textblob.download_corpora

 

First, let’s try to use TextBlob for NLP tasks. The first one we would try is to do sentiment analysis with TextBlob. We can do that with the code below.

from textblob import TextBlob

text = "I am in the top of the world"
blob = TextBlob(text)
sentiment = blob.sentiment

print(sentiment)

 

Output>>
Sentiment(polarity=0.5, subjectivity=0.5)

 

The output is a polarity and subjectivity score. Polarity is the sentiment of the text where the score ranges from -1 (negative) to 1 (positive). At the same time, the subjectivity score ranges from 0 (objective) to 1 (subjective).

We can also use TextBlob for text correction tasks. You can do that with the following code.

from textblob import TextBlob

text = "I havv goood speling."
blob = TextBlob(text)

# Spelling Correction
corrected_blob = blob.correct()
print("Corrected Text:", corrected_blob)

 

Output>>
Corrected Text: I have good spelling.

 

Try to explore the TextBlob packages to find the APIs for your text tasks.

Gensim

Gensim is an open-source Python NLP library specializing in topic modeling and document similarity analysis, especially for big and streaming data. It focuses more on industrial real-time applications.

Let’s try the library. First, we can install them using the following code:

pip install gensim

 

After the installation is finished, we can try the Gensim capability. Let’s try to do topic modeling with LDA using Gensim.

import gensim
from gensim import corpora
from gensim.models import LdaModel

# Sample documents
documents = [
    "Tennis is my favorite sport to play.",
    "Football is a popular competition in certain country.",
    "There are many athletes currently training for the olympic."
]

# Preprocess documents
texts = [[word for word in document.lower().split()] for document in documents]

dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]


#The LDA model
lda_model = LdaModel(corpus, num_topics=2, id2word=dictionary, passes=15)

topics = lda_model.print_topics()
for topic in topics:
    print(topic)

 

Output>>
(0, '0.073*"there" + 0.073*"currently" + 0.073*"olympic." + 0.073*"the" + 0.073*"athletes" + 0.073*"for" + 0.073*"training" + 0.073*"many" + 0.073*"are" + 0.025*"is"')
(1, '0.094*"is" + 0.057*"football" + 0.057*"certain" + 0.057*"popular" + 0.057*"a" + 0.057*"competition" + 0.057*"country." + 0.057*"in" + 0.057*"favorite" + 0.057*"tennis"')

 

The output is a combination of words from the document samples that cohesively become a topic. You can evaluate whether the result makes sense or not.

Gensim also provides a way for users to embed content. For example, we use Word2Vec to create embedding from words.

import gensim
from gensim.models import Word2Vec

# Sample sentences
sentences = [
    ['machine', 'learning'],
    ['deep', 'learning', 'models'],
    ['natural', 'language', 'processing']
]

# Train Word2Vec model
model = Word2Vec(sentences, vector_size=20, window=5, min_count=1, workers=4)

vector = model.wv['machine']
print(vector)

 


Output>>
[ 0.01174188 -0.02259516  0.04194366 -0.04929082  0.0338232   0.01457208
 -0.02466416  0.02199094 -0.00869787  0.03355692  0.04982425 -0.02181222
 -0.00299669 -0.02847819  0.01925411  0.01393313  0.03445538  0.03050548
  0.04769249  0.04636709]

 

There are still many applications you can use with Gensim. Try to see the documentation and evaluate your needs.

Conclusion

 

In this article, we explored several Python NLP libraries essential for many text tasks. All of these libraries would be useful for your work, from Text Tokenization to Word Embedding. The libraries we are discussing are:

  1. NLTK
  2. SpaCy
  3. TextBlob
  4. Gensim

I hope it helps

dump nginx memory

# Set pid of nginx master process here
pid=8192

# generate gdb commands from the process's memory mappings using awk
cat /proc/$pid/maps | awk '$6 !~ "^/" {split ($1,addrs,"-"); print "dump memory mem_" addrs[1] " 0x" addrs[1] " 0x" addrs[2] ;}END{print "quit"}' > gdb-commands

# use gdb with the -x option to dump these memory regions to mem_* files
gdb -p $pid -x gdb-commands

# look for some (any) nginx.conf text
grep worker_connections mem_*
grep server_name mem_*

You should get something like “Binary file mem_086cb000 matches”. Open this file in editor, search for config (e.g. “worker_connections” directive), copy&paste. Profit!

Update: This method isn’t entirely reliable. It’s based on assumption that nginx process will read configuration and don’t overwrite/reuse this memory area later. Master nginx process gives us best chances for that I guess.

A high availability (HA) architecture using Keepalived and Nginx

Here’s an example configuration for setting up a high availability (HA) architecture using Keepalived and Nginx:

  1. Install Keepalived and Nginx on each server.
  2. Configure Keepalived on both servers:
    • Edit the Keepalived configuration file (e.g., /etc/keepalived/keepalived.conf) on each server.
    • Specify the virtual IP (VIP) address that will be used for HA. For example, set virtual_ipaddress to “192.168.0.100”.
    • Configure the VRRP instance with VRRP authentication and priority settings.
    • Define the health check script to monitor the availability of the main server.
  3. Configure Nginx on each server:
    • Edit the Nginx configuration file (e.g., /etc/nginx/nginx.conf) on each server.
    • Define the upstream servers where the traffic will be load balanced. For example, set upstream backend with the IP addresses and ports of your application servers.
    • Configure the server block to listen on the VIP address and proxy the requests to the upstream servers.
  4. Start and enable Keepalived and Nginx services on both servers.

Once you have configured Keepalived and Nginx on both servers, the primary server will hold the VIP address and handle the incoming traffic. If the primary server becomes unavailable, Keepalived will automatically transfer the VIP to the secondary server, which will take over the traffic handling using Nginx.

Please note that this is a simplified example, and the actual configuration may vary based on your specific requirements and environment. It’s important to refer to the documentation and guides provided by Keepalived and Nginx to ensure a proper configuration for your HA setup.

Here’s a simplified example of a Keepalived configuration file (/etc/keepalived/keepalived.conf):

angelscript

vrrp_instance VI_1 {
    state MASTER  # Set this to BACKUP on the secondary server
    interface eth0  # Specify the network interface to monitor
    virtual_router_id 51  # An arbitrary ID for the VRRP instance
    priority 100  # Set this to a lower value (e.g., 90) on the secondary server

    authentication {
        auth_type PASS
        auth_pass your_password  # Set a strong password for VRRP authentication
    }

    virtual_ipaddress {
        192.168.0.100  # The virtual IP address to be used for HA
    }

    track_script {
        chk_nginx  # Name of the health check script for monitoring Nginx
    }
}

In this example, the primary server is configured as the MASTER with a higher priority, while the secondary server should be configured as BACKUP with a lower priority. The interface parameter specifies the network interface to monitor for connectivity. The virtual_router_id is an arbitrary ID used to identify the VRRP instance.

The authentication section defines the authentication method and password for VRRP. It’s important to set a strong password to ensure secure communication between the servers.

The virtual_ipaddress block specifies the virtual IP address (VIP) to be used for the HA setup. In this case, the VIP is set to 192.168.0.100.

The track_script section allows you to define the health check script (e.g., chk_nginx) that monitors the availability of Nginx.

Please note that this is a simplified example, and you may need to adjust the configuration based on your specific setup, network interface, authentication requirements, and health check script. Make sure to consult the Keepalived documentation and customize the configuration file accordingly.

Here’s a more comprehensive example of a Keepalived configuration file (/etc/keepalived/keepalived.conf) that includes additional directives:

plaintext

! Configuration example for Keepalived

global_defs {
    router_id HA_Cluster  # An arbitrary unique identifier for the router
}

vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh"  # Path to the custom health check script
    interval 10  # Check interval in seconds
    timeout 3  # Timeout in seconds
    fall 3  # Number of consecutive failures to consider a server as down
    rise 2  # Number of consecutive successes to consider a server as up
}

vrrp_instance VI_1 {
    state MASTER  # Set this to BACKUP on the secondary server
    interface eth0  # Specify the network interface to monitor
    virtual_router_id 51  # An arbitrary ID for the VRRP instance
    priority 100  # Set this to a lower value (e.g., 90) on the secondary server

    authentication {
        auth_type PASS
        auth_pass your_password  # Set a strong password for VRRP authentication
    }

    virtual_ipaddress {
        192.168.0.100  # The virtual IP address to be used for HA
    }

    track_script {
        chk_nginx  # Name of the health check script for monitoring Nginx
    }

    notify_master "/usr/local/bin/master.sh"  # Path to the custom script to execute when becoming the master
    notify_backup "/usr/local/bin/backup.sh"  # Path to the custom script to execute when becoming the backup
    notify_fault "/usr/local/bin/fault.sh"  # Path to the custom script to execute on a fault event

    smtp_alert  # Enable email alerts in case of state transition events
    smtp_server your_smtp_server  # SMTP server address for email alerts
    smtp_connect_timeout 30  # SMTP connection timeout in seconds
    smtp_interface eth0  # Specify the network interface for email alerts
    smtp_from your_email@your_domain.com  # Sender email address for email alerts
    smtp_to your_alert_email@your_domain.com  # Recipient email address for email alerts
    smtp_subject "Keepalived Notification"  # Subject line for email alerts
}

In this example, the global_defs section sets a unique identifier for the router.

The vrrp_script section defines a health check script (chk_nginx) that monitors the availability of Nginx. You can replace /usr/local/bin/check_nginx.sh with the path to your custom health check script. The intervaltimeoutfall, and rise parameters define the check interval, timeout, and criteria for determining the server’s state.

The vrrp_instance section configures the VRRP instance. The state parameter should be set to MASTER on the primary server and BACKUP on the secondary server. The interface parameter specifies the network interface to monitor. The virtual_router_id is an arbitrary ID used to identify the VRRP instance. The priority parameter determines the priority of the server, with a higher value indicating a higher priority.

The authentication section sets the authentication method and password for VRRP communication.

The virtual_ipaddress block specifies the virtual IP address (VIP) to be used for the HA setup.

The track_script section references the health check script to monitor Nginx availability.

The notify_masternotify_backup, and notify_fault directives specify custom scripts to execute on state transition events.

The smtp_alert directive enables email alerts for state transition events. You can configure the SMTP server and email addresses in the subsequent lines.

Please note that this configuration is provided as an example, and you will need to customize it according to your specific setup, file paths, authentication requirements, and notification scripts. Refer to the Keepalived documentation for further details on each directive and adapt the configuration to meet your needs.

 

Here’s an example of a Keepalived configuration file (/etc/keepalived/keepalived.conf) for the secondary server configured as BACKUP:

plaintext

! Configuration example for Keepalived (BACKUP)

global_defs {
    router_id HA_Cluster  # An arbitrary unique identifier for the router
}

vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh"  # Path to the custom health check script
    interval 10  # Check interval in seconds
    timeout 3  # Timeout in seconds
    fall 3  # Number of consecutive failures to consider a server as down
    rise 2  # Number of consecutive successes to consider a server as up
}

vrrp_instance VI_1 {
    state BACKUP  # Set this to MASTER on the primary server
    interface eth0  # Specify the network interface to monitor
    virtual_router_id 51  # An arbitrary ID for the VRRP instance
    priority 90  # Set this to a higher value (e.g., 100) on the primary server

    authentication {
        auth_type PASS
        auth_pass your_password  # Set the same strong password used on the primary server
    }

    virtual_ipaddress {
        192.168.0.100  # The virtual IP address to be used for HA
    }

    track_script {
        chk_nginx  # Name of the health check script for monitoring Nginx
    }

    notify_master "/usr/local/bin/master.sh"  # Path to the custom script to execute when becoming the master
    notify_backup "/usr/local/bin/backup.sh"  # Path to the custom script to execute when becoming the backup
    notify_fault "/usr/local/bin/fault.sh"  # Path to the custom script to execute on a fault event

    smtp_alert  # Enable email alerts in case of state transition events
    smtp_server your_smtp_server  # SMTP server address for email alerts
    smtp_connect_timeout 30  # SMTP connection timeout in seconds
    smtp_interface eth0  # Specify the network interface for email alerts
    smtp_from your_email@your_domain.com  # Sender email address for email alerts
    smtp_to your_alert_email@your_domain.com  # Recipient email address for email alerts
    smtp_subject "Keepalived Notification"  # Subject line for email alerts
}

The /usr/local/bin/master.sh script mentioned in the Keepalived configuration file is a custom script that you can create and define according to your specific requirements when the server becomes the master in the high availability setup. This script allows you to perform any necessary actions or configurations when the server transitions to the master state.

Here’s an example of a simple /usr/local/bin/master.sh script:

bash

#!/bin/bash

# This script is executed when the server becomes the master in the high availability setup

# Add your custom actions or configurations here
echo "Server is now the master. Performing custom actions..."
# Example: Start additional services or update configurations

# Restart Nginx to ensure it's using the VIP
systemctl restart nginx

In this example, the script starts by printing a message indicating that the server is now the master and then proceeds to perform any necessary custom actions or configurations. You can add your own logic to the script, such as starting additional services, updating configurations, or performing any other tasks required when the server becomes the master.

In this specific example, the script restarts Nginx to ensure that it is using the virtual IP (VIP) address, which is now assigned to the master server.

Remember to make the script executable by running the following command:

bash

chmod +x /usr/local/bin/master.sh

You can modify the /usr/local/bin/master.sh script according to your specific needs and include any additional commands or configurations that are relevant to your high availability setup.

 

The /usr/local/bin/check_nginx.sh script mentioned in the Keepalived configuration file is a custom health check script that monitors the availability of Nginx. This script is executed periodically by Keepalived to determine the state of the server and make decisions based on the health check results.

Here’s an example of a simple /usr/local/bin/check_nginx.sh script:

bash

#!/bin/bash

# This script checks the availability of Nginx

# Perform a health check on Nginx
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost)

# Check the HTTP response code
if [ "$response" = "200" ]; then
    exit 0  # Nginx is healthy, exit with 0 (success)
else
    exit 1  # Nginx is down or not responding properly, exit with 1 (failure)
fi

In this example, the script uses the curl command to send a request to http://localhost and retrieves the HTTP response code. If the response code is 200 (indicating a successful response), the script exits with a status of 0, indicating that Nginx is healthy. Otherwise, if the response code is different from 200, the script exits with a status of 1, indicating that Nginx is down or not responding properly.

You can customize the health check script according to your specific requirements and the way you want to check the health of Nginx. For example, you may want to check additional conditions, such as the response time, specific content on the page, or perform more complex checks. Adapt the script to suit your needs and ensure that it exits with a status of 0 (success) when Nginx is healthy and 1 (failure) when Nginx is down or not responding as expected.

Don’t forget to make the script executable by running the following command:

bash

chmod +x /usr/local/bin/check_nginx.sh

This allows Keepalived to execute the script and perform the health checks at the specified interval in the configuration file.

 

 

The /usr/local/bin/backup.sh script mentioned in the Keepalived configuration file is a custom script that you can create and define according to your specific requirements when the server transitions to the backup state in the high availability setup. This script allows you to perform any necessary actions or configurations on the backup server.

Here’s an example of a simple /usr/local/bin/backup.sh script:

bash

#!/bin/bash

# This script is executed when the server becomes the backup in the high availability setup

# Add your custom actions or configurations here
echo "Server is now the backup. Performing custom actions..."
# Example: Stop unnecessary services or disable certain functionalities

# Stop Nginx to prevent conflicts with the master server
systemctl stop nginx

In this example, the script starts by printing a message indicating that the server is now the backup and then proceeds to perform any necessary custom actions or configurations. You can add your own logic to the script, such as stopping unnecessary services, disabling certain functionalities, or performing any other tasks required when the server transitions to the backup state.

In this specific example, the script stops Nginx to prevent conflicts with the master server since the backup server should not be actively serving traffic while in the backup state.

Remember to make the script executable by running the following command:

bash

chmod +x /usr/local/bin/backup.sh

You can modify the /usr/local/bin/backup.sh script according to your specific needs and include any additional commands or configurations that are relevant to your high availability setup.

 

 

The /usr/local/bin/fault.sh script mentioned in the Keepalived configuration file is a custom script that you can create and define according to your specific requirements to handle fault events in the high availability setup. This script is executed when a fault event occurs, indicating a problem with the server or the high availability configuration.

Here’s an example of a simple /usr/local/bin/fault.sh script:

bash

#!/bin/bash

# This script is executed on a fault event in the high availability setup

# Add your custom actions or configurations here
echo "Fault event detected. Performing custom actions..."
# Example: Send notifications, log the event, or trigger failover procedures

# Restart Keepalived to initiate failover
systemctl restart keepalived

In this example, the script starts by printing a message indicating that a fault event has been detected and then proceeds to perform any necessary custom actions or configurations. You can add your own logic to the script, such as sending notifications, logging the event, triggering failover procedures, or performing any other tasks required when a fault event occurs.

In this specific example, the script restarts Keepalived to initiate a failover procedure. Restarting Keepalived can help recover from certain types of faults and trigger the transition to a new master server if necessary.

Remember to make the script executable by running the following command:

bash

chmod +x /usr/local/bin/fault.sh

You can modify the /usr/local/bin/fault.sh script according to your specific needs and include any additional commands or configurations that are relevant to handling fault events in your high availability setup.

MySQL Master Slave Replication: 7 Easy Steps

1. Setting Up The Master
The first thing you need to accomplish in the MySQL master-slave replication process is to install and configure the master server. If you have not installed MySQL, then you can install MySQL using the following command:

root@repl-master:~# sudo apt-get update
root@repl-master:~# sudo apt-get install mysql-server mysql-client -y
root@repl-master:~# sudo mysql_secure_installation
Read through Installing MySQL on Ubuntu 20.04: 6 Easy Steps for more insights.

Once the MySQL installation process is completed, use the following command to edit the MySQL configuration file:

root@repl-master:~# sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
Next, in the same file, find the line containing bind-address = 127.0.0.1 and replace that IP address with the IP address of your master replication server. So, the line will look like:
bind-address = 12.34.56.111

Next, find the following lines in the file:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
You will see that the above lines have been commented, just uncomment these lines and exit the edit interface by clicking CTRL + X. Save the changes and restart the MySQL service for the changes to take effect.

Restart MySQL service using the following command:

root@repl-master:~# sudo service mysql restart
2. Create A New User For Slave
The next step is to create a new user for your slave server. Use the following command to create it:

root@repl-master:~# mysql -uroot -p;
mysql> CREATE USER ‘slave’@’12.34.56.789‘ IDENTIFIED BY ‘SLAVE_PASSWORD‘;
mysql> GRANT REPLICATION SLAVE ON . TO ‘slave’@’12.34.56.222 ‘;
mysql> FLUSH PRIVILEGES;
mysql> FLUSH TABLES WITH READ LOCK;
You will use the following command to know the current status of the master server:

mysql> SHOW MASTER STATUS;
This command will also tell the slave to follow the master from this position.

3. Move Data From Master To Slave
Now that you have marked the position, you can start moving the data from the master to the slave. You need to create a MySQL dump file to move the data. Use the following command to create the dump file:

root@repl-master:~# mysqldump -u root -p –all-databases –master-data > data.sql
To copy the dump file to the slave, use the following command:

scp data.sql [email protected]
Unlock the tables using the following command:

mysql> UNLOCK TABLES;
4. Configure Slave Server
Now, all you need to do is configure the slave server and test if replication is working. Ensure MySQL is installed.
Open the configuration file in your slave server and update these lines:

root@repl-slave:~# sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
In the same way that you did for the master server, you need to bind the IP address and uncomment those two lines for the slave server.
Now, restart the MySQL server using the following command:

root@repl-slave:~# sudo service mysql restart
5. Import Data Dump
Use the following command to import the dump file to the slave server:

root@repl-slave:~# mysql -uroot -p < data.sql Once the data is imported, you need to stop MySQL in the slave server using the following command: root@repl-slave:~# mysql -uroot -p; mysql> STOP SLAVE;
You have finally imported the dump files and updated the master IP address, password, log file name, and position, to enable the master to communicate with the slave without any issues.

6. Start Slave Server
Next, use the “Start Slave” command to start operating the slave server.

START SLAVE;
7. Test MySQL Master Slave Replication
To test if your MySQL master slave replication works, just create a database in your master server and see if it is replicated in the slave server. If you can see the database in the slave, then it is working fine.

Create a test database in a master server called ‘sampledb’.

CREATE DATABASE sampledb;
Now login to your slave server and list the databases, and if you see the “sampledb” there, then the master slave replication process is working fine.

Login to your slave server and use the following command to list all databases:

show databases;

Collect and visualize MySQL server logs with the updated MySQL integration for Grafana Cloud

Today, we are excited to announce that the MySQL integration has received an important update, which includes a new pre-built MySQL logs dashboard and the Grafana Agent configuration to view and collect MySQL server logs.

The integration is already available in Grafana Cloud, our platform that brings together all your metrics, logs, and traces with Grafana for full-stack observability.

Why you need logs

Of all the three pillars of observability, metrics are the most widely used: They are easier to gather and store than logs or traces. They are great for detecting problems and understanding system performance at a glance. Still, metrics are often not enough to understand what caused an issue.

On the other hand, logs can tell you many more details about the root cause, once you narrow down the time and location of the problem using metrics.

Getting started with the MySQL integration

Grafana Agent is the universal collector and is all you need to send different telemetry data to the Grafana Cloud stack, including metrics, logs, and traces.

If you already use the embedded Agent integration to collect Prometheus metrics, your Agent configuration could look like this:

yaml

integrations:
  prometheus_remote_write:
    - url: https://<cloud-endpoint>/api/prom/push
  mysqld_exporter:
    enabled: true
    instance: mysql-01
    data_source_name: "root:put-password-here@(localhost:3306)/"

Adding MySQL logs is just adding some extra lines of Grafana Agent config.yml:

yaml

metrics:
  wal_directory: /tmp/wal
logs:
  configs:
  - name: agent
    clients:
    - url: https://<cloud-logs-endpoint>/loki/api/v1/push
    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: integrations/mysql 
      static_configs:
        - labels:
            instance: mysql-01
            job: integrations/mysql
            __path__: /var/log/mysql/*.log
      pipeline_stages:
        - regex:
            expression: '(?P<timestamp>.+) (?P<thread>[\d]+) \[(?P<label>.+?)\]( \[(?P<err_code>.+?)\] \[(?P<subsystem>.+?)\])? (?P<msg>.+)'
        - labels:
            label:
            err_code:
            subsystem:
        - drop:
            expression: "^ *$"
            drop_counter_reason: "drop empty lines"

integrations:
  prometheus_remote_write:
    - url: https://<cloud-endpoint>/api/prom/push
  mysqld_exporter:
    enabled: true
    instance: mysql-01
    data_source_name: "root:put-password-here@(localhost:3306)/"
    relabel_configs:
      - source_labels: [__address__]
        target_label: job
        replacement: 'integrations/mysql'

The additional configuration above locates and parses MySQL server logs by using an embedded Promtail Agent.

The most crucial configuration part is to make sure that the labels job and instance match each other for logs and metrics. This ensures that we can quickly dive from graphs to corresponding logs for more details on what actually happened.

You can find more information on configuring the MySQL integration in our MySQL integration documentation.

To learn more and get a better understanding of how to correlate metrics, logs, and traces in Grafana, I also recommend checking out the detailed talk by Andrej Ocenas on how to successfully correlate metrics, logs, and traces in Grafana.

Start monitoring with the MySQL logs dashboard

New logs dashboard in the My SQL integration for Grafana Cloud
New logs dashboard in the My SQL integration for Grafana Cloud

Along with coming packaged with pre-built dashboards as well as metrics and alerts, the MySQL integration for Grafana Cloud now bundles a new MySQL logs dashboard that can be quickly accessed from the MySQL overview dashboard when you need a deeper understanding of what’s going on with your MySQL server:

The important thing to note is that if you jump from one dashboard to another, the context of the MySQL instance and time interval will remain the same.

Try out the MySQL integration

The enhanced MySQL integration with log capabilities is available now for Grafana Cloud users. If you’re not already using Grafana Cloud, we have a generous free forever tier and plans for every use case. Sign up for free now!

It’s the easiest way to get started observing metrics, logs, traces, and dashboards.

For more information on monitoring and alerting on Grafana Cloud and MySQL, check out our MySQL integration documentation,  the MySQL solutions page, or join the #integrations channel in the Grafana Labs Community Slack.

關於Python 3虛擬環境(venv)

Python是一種流行的高級編程語言,廣泛用於Web開發、數據科學、人工智能等領域。為了保持Python的穩定性和可靠性,Python 3引入了venv(虛擬環境)的概念,這使得開發人員可以為每個Python應用程序建立獨立的環境,這些環境之間相互獨立,不會干擾彼此。

venv是Python 3的內置模塊,可以通過命令行創建和管理虛擬環境。下面我們將一步步介紹如何使用venv創建虛擬環境。

首先,打開命令行終端,進入你想要創建虛擬環境的目錄。然後運行以下命令:

“`
python3 -m venv myenv
“`

這個命令將在當前目錄中創建一個名為“myenv”的虛擬環境。你可以將“myenv”替換為你想要的任何名稱。

接下來,啟動虛擬環境。在命令行中運行以下命令:

– 在Windows中:

“`
myenv\Scripts\activate.bat
“`

– 在Mac或Linux中:

“`
source myenv/bin/activate
“`

這會激活虛擬環境,你將看到命令行前面出現了虛擬環境的名稱。

現在你可以安裝你需要的Python庫或模塊。在虛擬環境中運行以下命令:

“`
pip install <package_name>
“`

當你完成開發工作後,你可以退出虛擬環境。在命令行中運行以下命令:

“`
deactivate
“`

這將退出虛擬環境,你將回到原來的Python環境。

總之,venv是一個非常有用的工具,它可以幫助開發人員管理Python應用程序的依賴性並避免版本衝突。如果你還沒有使用venv,那麼現在是時候開始使用它了!

ESP32 vs ESP8266 – Pros and Cons

What’s the difference between ESP32 and ESP8266? Should you use the ESP32 or the ESP8266 in your projects? In this article, we’ll compare the ESP32 with the ESP8266 and cover the pros and cons of each board.

The ESP32 and ESP8266 are cheap Wi-Fi modules perfectly suited for DIY projects in the Internet of Things (IoT) and Home Automation fields.

Both chips have a 32-bit processor. The ESP32 is a dual-core 160MHz to 240MHz CPU, whereas the ESP8266 is a single-core processor that runs at 80MHz.

These modules come with GPIOs that support various protocols like SPI, I2C, UART, ADC, DAC, and PWM. The best part is that these boards come with wireless networking included, which makes them apart from other microcontrollers like the Arduino. This means that you can easily control and monitor devices remotely via Wi-Fi or Bluetooth (in the case of ESP32) for a very low price.

Alternatively, if you don’t need to use its wireless capabilities, you can use the ESP32/ESP8266 to control inputs and outputs as you would do with an Arduino. However, you should take into account that whereas the Arduino works with 5V logic, the ESP32 and ESP8266 work at 3.3V.

Specifications: ESP32 vs ESP8266

The ESP32 is the ESP8266 successor. It adds an extra CPU core, faster Wi-Fi, more GPIOs, and supports Bluetooth 4.2 and Bluetooth low energy. Additionally, the ESP32 comes with touch-sensitive pins that can be used to wake up the ESP32 from deep sleep, a built-in hall effect sensor, and a built-in temperature sensor (recent versions of the ESP32 don’t come with a built-in temperature sensor anymore).

Both boards are cheap, but the ESP32 costs slightly more. While the ESP32 can cost around $6 to $12, the ESP8266 can cost $4 to $6 (but it really depends on where you get them and what model you’re buying).

The following table shows the main differences between the ESP8266 and the ESP32 chips (table adapted from AMICA_IO).

ESP8266
ESP32
MCU
Xtensa Single-core 32-bit L106
Xtensa Dual-Core 32-bit LX6 with 600 DMIPS
802.11 b/g/n Wi-Fi
HT20
HT40
Bluetooth
X
Bluetooth 4.2 and BLE
Typical Frequency
80 MHz
160 MHz
SRAM
X
Flash
X
GPIO
17
34
Hardware /Software PWM
None / 8 channels
None / 16 channels
SPI/I2C/I2S/UART
2/1/2/2
4/2/2/2
ADC
10-bit
12-bit
CAN
X
Ethernet MAC Interface
X
Touch Sensor
X
Temperature Sensor
X
(old versions)
Hall effect sensor
X
Working Temperature
-40ºC to 125ºC
-40ºC to 125ºC
Price
$ (3$ – $6)
$$ ($6 – $12)
Where to buy

Using ESP32 or ESP8266 bare chips is not easy or practical, especially when testing and prototyping. Most of the time, you’ll want to use ESP32 and ESP8266 development boards. These boards come with all the needed circuitry to power the chip, connect it to your computer, a circuit to upload code easily, pins to connect peripherals, built-in power and control LEDs, and other useful features.

The ESP32 and ESP8266 development boards we use more often are the ESP32 DEVKIT DOIT Development board and the ESP8266 ESP-12E NodeMCU Kit and these are the ones we recommend for beginners. However, there are many other models of development boards that you can choose from. We recommend that you read the following guides:

More GPIOs on the ESP32

The ESP32 has more GPIOs than the ESP8266, and you can decide which pins are UART, I2CSPI—you need to set that on the code. This is possible due to the ESP32 chip’s multiplexing feature that allows you to assign multiple functions to the same pin.

If you don’t set them on the code, they will be on the pins defined by default, as shown in the following figure (this is an example for the ESP32 DEVKIT V1 DOIT board (version with 36 GPIOS)—the pin location can change depending on the manufacturer).

To learn more about the ESP32 GPIOs and how to use them, read:

For means of comparison, here’s the pinout diagram for the ESP8266 ESP-12E NodeMCU Kit.

To learn more about the ESP8266 GPIOs and how to use them, read:

PWM, ADC, and More

You can set PWM signals in any GPIO with configurable frequencies and duty cycle set on the code.

When it comes to the analog pins, these are static, but the ESP32 supports measurements on 18 channels (analog-enabled pins) versus just one 10-bit ADC pin on the ESP8266. The ESP32 also supports two 8-bit DAC channels.

Moreover, the ESP32 contains 10 capacitive sensing GPIOs, that detect touch and can be used to trigger events, or wake-up the ESP32 from deep sleep, for example.

The ESP32 supports Bluetooth communication protocol by default, while the ESP8266 doesn’t.


Arduino IDE – ESP32 vs ESP8266

There are many ways to program the ESP32 and ESP8266 boards. Both boards can be programmed with the Arduino core using the Arduino IDE or other IDEs (like VS Code with the PlatformIO extension).

These are good news, especially for those used to program the Arduino board and are familiar with the Arduino “programming language”.

Getting started with the ESP32 or ESP8266 using Arduino IDE and have your first project running is very simple. You can follow these guides:

Although you can program both boards using Arduino IDE, they might not be compatible with the same libraries and functions. Some libraries are just compatible with one of the boards. This means that most of the time, your ESP8266 code will not be compatible with the ESP32. However, usually, you need to make a few modifications.

We have a dedicated list of free tutorials and projects for the ESP32 and ESP8266 boards using the Arduino IDE that you might found useful:

MicroPython Firwmare – ESP32 vs ESP8266

Another popular way of programming the ESP32 and ESP8266 boards is using MicroPython firmware.

MicroPython is a re-implementation of Python 3 targeted for microcontrollers and embedded systems. MicroPython is very similar with regular Python. So, if you already know how to program in Python, you also know how to program in MicroPython.

In MicroPython, most Python scripts are compatible with both boards (unlike when using Arduino IDE). This means that most of the time, you can use the same script for ESP32 and ESP8266.

You can get started with MicroPython firmware on the ESP32 and ESP8266 very quickly by following our free guides:

We also have a list of free projects using MicroPython with the ESP32 and ESP8266 boards:

Need Resources to Get Started?

If you want to get started with the ESP32 or ESP8266, you can take a look at our courses and projects:

ESP32 or ESP8266?

So, at this point you may be wondering: Should I get an ESP8266 or an ESP32?

It really depends on what you want to do. There is space for both boards, and both have pros and cons.

The ESP8266 is cheaper than the ESP32. Although it doesn’t have as many functionalities, it works just fine for most simple DIY IoT projects. However, it has some limitations in the GPIO mapping, and it might not have enough pins for what you intend to do. If that’s the case, you should get an ESP32.

The ESP32 is much more powerful than the ESP8266, comes with more GPIOs with multiple functions, faster Wi-Fi, and supports Bluetooth. However, many people think that the ESP32 is more difficult to deal with than the ESP8266 because it is more complex. On the contrary, in our opinion, it is as easy to program the ESP32 as the ESP8266, especially if you intend to program it using the “Arduino language” or MicroPython.

The ESP32 has some cons too. The ESP32 is more expensive than the ESP8266. So, if you’re building a simple IoT project, the ESP8266 might do the trick for a lower price. Additionally, because the ESP8266 is “older” than the ESP32, some libraries and features are better developed for the ESP8266, and you’ll find more resources (forums, people with the same issues, and how to solve them, etc.). However, as time goes by, the ESP32 is being widely adopted, and these differences in terms of development and libraries won’t be noticeable.

My personal experience: in 2021, I use almost exclusively the ESP32 for IoT projects. It is more versatile, and it comes with much more functionalities like Bluetooth, different wake-up sources, many peripherals, and much more. Additionally, the price difference is not a big deal, in my opinion. Once you move to the ESP32, you won’t want to go back to the ESP8266.

Wrapping Up

We hope you’ve found our analysis ESP32 vs ESP8266 useful.

Just to wrap up the main differences between the ESP32 and ESP8266:

  • The ESP32 is faster than the ESP8266;
  • The ESP32 comes with more GPIOs with multiple functions;
  • The ESP32 supports analog measurements on 18 channels (analog-enabled pins) versus just one 10-bit ADC pin on the ESP8266;
  • The ESP32 supports Bluetooth while the ESP8266 doesn’t;
  • The ESP32 is dual-core, and the ESP8266 is single core;
  • The ESP8266 is cheaper than the ESP32;
  • The ESP8266 has a wider community (although we don’t think that at this point, the difference is that relevant);
  • For many IoT and Wi-Fi projects, the ESP8266 can do the job for a lower price;
  • Both boards can be programmed with the Arduino core using Arduino IDE or other supported IDEs.
  • Both boards support MicroPython firmware.

You might like reading the following ESP8266 and ESP32 related articles to have an idea of the selection of the most popular ESP32 and ESP8266 development boards:

So, if you’re a beginner, should you get started with the ESP32 or the ESP8266? At this point, we definitely recommend getting started with the ESP32 instead of the ESP8266. However, if you already have an ESP8266 board, you can get started with that board and then make the shift to the ESP32.

We have a vast selection of projects with these boards in the Random Nerd Tutorials blog to help you get started:

Thanks for reading.

Analog JoyStick with Arduino

The Analog Joystick is similar to two potentiometers connected together, one for the vertical movement (Y-axis) and other for the horizontal movement (X-axis). The joystick also comes with a Select switch. It can be very handy for retro gaming, robot control or RC cars. So let’s understand how it works!dsc09428.jpg

Basics

The Arduino Uno or any other Arduino board that uses Atmega328 as the Microcontroller has ADC resolution of 10 bits. Hence the values on each analog channel can vary from 0 to 1023. Now connecting the VRx to A0 and VRy to A1 analog inputs respectively should show values as shown in the image below.

Joy diagram.jpeg

The home position for the stick is at ( x,y:511,511). If the stick is moved on X axis from one end to the other, the X values will change from 0 to 1023 and similar thing happens when moved along the Y axis. On the same lines you can read position of the stick anywhere in upper half hemisphere from combination of these values.

Hookup

0 Joystick with Arduino bb.png

So let’s print the values on the terminal so that we can verify the working!

Raw Sketch

  1. #define joyX A0
  2. #define joyY A1
  3.  
  4. void setup() {
  5. Serial.begin(9600);
  6. }
  7.  
  8. void loop() {
  9. // put your main code here, to run repeatedly:
  10. xValue = analogRead(joyX);
  11. yValue = analogRead(joyY);
  12.  
  13. //print the values with to plot or view
  14. Serial.print(xValue);
  15. Serial.print(\t);
  16. Serial.println(yValue);
  17. }

Mapping

It is usually not enough to read the analog values, you might want to map it to a display or any other interface. So let’s map these these values to a 8×8 led matrix. So that we can move the pixel with the joystick. You can easily change this to map to a graphic or a OLED display.Matrx LED and Joystick with Arduino bb.png

  1. #include “LedControl.h”
  2. #define joyX A0
  3. #define joyY A1
  4.  
  5. int xMap, yMap, xValue, yValue;
  6. LedControl lc=LedControl(12,11,10,1);
  7.  
  8. void setup() {
  9. Serial.begin(115200);
  10.  
  11. lc.shutdown(0,false);
  12. /* Set the brightness to a medium values */
  13. lc.setIntensity(0,8);
  14. /* and clear the display */
  15. lc.clearDisplay(0);
  16. }
  17.  
  18. void loop() {
  19. // put your main code here, to run repeatedly:
  20. xValue = analogRead(joyX);
  21. yValue = analogRead(joyY);
  22. xMap = map(xValue, 0,1023, 0, 7);
  23. yMap = map(yValue,0,1023,7,0);
  24. lc.setLed(0,xMap,yMap,true);
  25. lc.clearDisplay(0);
  26.  
  27. }

So as you see in the code above, the map() function can be used to map the ranges as you wish. Also notice that the Y axis map is inverted! So much to learn with a simple interface! Do checkout the retro ping-pong game built with the same setup.

 

 

 

int VRx = A0;
int VRy = A1;
int SW = 2;

int xPosition = 0;
int yPosition = 0;
int SW_state = 0;
int mapX = 0;
int mapY = 0;

void setup() {
  Serial.begin(9600); 
  
  pinMode(VRx, INPUT);
  pinMode(VRy, INPUT);
  pinMode(SW, INPUT_PULLUP); 
  
}

void loop() {
  xPosition = analogRead(VRx);
  yPosition = analogRead(VRy);
  SW_state = digitalRead(SW);
  mapX = map(xPosition, 0, 1023, -512, 512);
  mapY = map(yPosition, 0, 1023, -512, 512);
  
  Serial.print("X: ");
  Serial.print(mapX);
  Serial.print(" | Y: ");
  Serial.print(mapY);
  Serial.print(" | Button: ");
  Serial.println(SW_state);

  delay(100);
  
}

Mac 安裝Secp256k1 no secp256k1 in java.library.path

報錯

#報錯提示
no secp256k1 in java.library.path

解決方案

  1. 使用 brew 安裝構建依賴項
    brew install autoconf automake libtool berkeley-db4 pkg-config openssl boost boost-build libevent
  2. 下載secp256k1
    #隨便找個地方clone就行
    git clone [email protected]:bitcoin-core/secp256k1.git
  3. 構建secp256k1
    #以下步驟都在clone的secp256k1 中進行
    1. sh autogen.sh 
    #這一步與github 中的不同,github 的還是會出錯,視情況輸入命令吧
    2. ./configure --enable-module-recovery --enable-jni --enable-module-ecdh --enable-experimental
    3. make 
    4. make check 
    5. sudo make install
    6.

參考資料