23 Apr 2017
Adding code block in Jekyll blog should be easy. And though highlighting in markdown using back-ticks ``` worked but it didnt work as well as it should. The formatted java code was getting messed up as the lines were getting wrapped. I needed well formatted code with scrolling which should be easy to read.
This is where highlight.js came to rescue. There were other options and hacks but this was the clear winner because it was easy to integrate, supported many languages and comes with color themes.
- Only 1 step is needed. Add highlight js and css of the color scheme you want in the header page. Either you can add cdn url as shown below or copy it in your project and give local path. This will find and all highlight the code inside of <pre><code> tags
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.11.0/styles/default.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.11.0/highlight.min.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
-
highlight.js provide many color schemes and all you have to do is choose the right css and refer its path.
-
here is the list of cdn links of all color schemes
-
here is github link of all color schemes
-
here is the demo for all supported languages and schemes
01 Dec 2016
This guide explains the steps needed to secure Spring REST APIs with basic authentication. Once done, the clients would need to send Base64 encoded credentials with each request, using HTTP [Authorization] header.
- Add dependency in POM for spring security
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
<version>4.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
<version>4.1.1.RELEASE</version>
</dependency>
- Configure users with username, password and roles. Also define http urls which needs authentication along with the roles to whom urls would be accessible.
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.config.http.SessionCreationPolicy;
@Configuration
@EnableWebSecurity
public class SecurityConfiguration extends WebSecurityConfigurerAdapter {
public static String REALM = "REALM";
@Autowired
public void configureGlobalSecurity(AuthenticationManagerBuilder auth) throws Exception {
auth.inMemoryAuthentication().withUser("abc").password("abc123").roles("USER");
auth.inMemoryAuthentication().withUser("xyz").password("xyz123").roles("USER");
}
@Override
protected void configure(HttpSecurity http) throws Exception {
http.csrf().disable()
.authorizeRequests()
.antMatchers("/labels.json").hasRole("USER")
.and().httpBasic().realmName(REALM).authenticationEntryPoint(getBasicAuthEntryPoint())
.and().sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
}
@Bean
public CustomBasicAuthenticationEntryPoint getBasicAuthEntryPoint() {
return new CustomBasicAuthenticationEntryPoint();
}
@Override
public void configure(WebSecurity web) throws Exception {
web.ignoring().antMatchers(HttpMethod.OPTIONS, "/**");
}
}
- Configure BasicAuthenticationEntryPoint to return response in case of invalid/missing credentials. Since its a REST API, it makes sense to return some response and not redirect user to any login page.
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import static SecurityConfiguration.REALM;
public class CustomBasicAuthenticationEntryPoint extends BasicAuthenticationEntryPoint {
@Override
public void commence(final HttpServletRequest request,
final HttpServletResponse response,
final AuthenticationException authException) throws IOException, ServletException {
response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
response.addHeader("WWW-Authenticate", "Basic realm=" + getRealmName() + "");
PrintWriter writer = response.getWriter();
writer.println("HTTP Status 401 : " + authException.getMessage());
}
@Override
public void afterPropertiesSet() throws Exception {
setRealmName(REALM);
super.afterPropertiesSet();
}
}
- Enable spring security in web.xml. This can also be done by just creating a new class extending AbstractSecurityWebApplicationInitializer but I have faced issues with jetty using java config so I prefer xml config.
<filter>
<filter-name>springSecurityFilterChain</filter-name>
<filter-class>org.springframework.web.filter.DelegatingFilterProxy
</filter-class>
</filter>
<filter-mapping>
<filter-name>springSecurityFilterChain</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
28 Nov 2016
Problem: Sometimes mysql master and slave go out of sync, this could happen due to various reasons like master stops writing to binary logs, duplicate entry exists in slave or some other insert/update constraint failure in slave (this could happen when slave database is not read only and some clients connects directly to slave). In such scenarios, the only way to bring back master and slave in sync is to rebuild slave from master. Below are tried and tested steps to rebuild slave from master where the master-slave setup already exists.
Consider this scenario with master and slave server as given below:
master/slave
exdap211/exdap212
###Problem on slave
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: exdap211.xyz.in
Master_User: slave_sec
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000044
Read_Master_Log_Pos: 552489846
Relay_Log_File: mysqld-relay-bin.000041
Relay_Log_Pos: 390393576
Relay_Master_Log_File: mysql-bin.000044
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1452
Last_Error: Error 'Cannot add or update a child row: a foreign key constraint fails...'
Skip_Counter: 0
Exec_Master_Log_Pos: 390393413
Relay_Log_Space: 552490230
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1452
Last_SQL_Error: Error 'Cannot add or update a child row: a foreign key constraint fails...'
Replicate_Ignore_Server_Ids:
Master_Server_Id: 100412
Master_UUID: f20637be-3530-11e5-a860-005056a05b60
Master_Info_File: /var/db/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 160322 16:12:05
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
###Solution: Rebuild slave
On Master
mysql -u root -p (you will be prompted for password)
mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000044
Position: 553540493
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
mysql> reset master;
Query OK, 0 rows affected (1.40 sec)
- Check master status again
mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 13766
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
- Exit from mysql prompt and start taking dump of master. This will take some time depending on size of database
bring@mysql2qa:~$ mysqldump -uroot -p --all-databases --master-data=1 --single-transaction --quick > /tmp/dbdump.db
Enter password:
- Once completed, check the file to be sure
ls -ltrh
-rw-r--r-- 1 bring deploy 2.0G Mar 29 10:03 dbdump.db
###Done with steps on master, now login to slave
- SCP dump file from master to slave
scp user@exdap211:/var/db/backup/dbdump.db /var/db/backup/dbdump.db
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'exdap211,139.114.173.205' (RSA) to the list of known hosts.
dbdump.db 100% 1976MB 164.7MB/s 00:12
- Check the file to be sure
-rw-r--r-- 1 bring deploy 2.0G Mar 29 10:08 dbdump.db
- Now login to mysql, stop slave and then reset slave and then exit
mysql> stop slave;
Query OK, 0 rows affected (0.00 sec)
mysql> reset slave;
Query OK, 0 rows affected (0.25 sec)
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: exdap211.xyz.in
Master_User: slave_sec
Master_Port: 3306
Connect_Retry: 60
Master_Log_File:
Read_Master_Log_Pos: 4
Relay_Log_File: mysqld-relay-bin.000001
Relay_Log_Pos: 4
Relay_Master_Log_File:
Slave_IO_Running: No
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 0
Relay_Log_Space: 143
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 100412
Master_UUID: f20637be-3530-11e5-a860-005056a05b60
Master_Info_File: /var/db/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
mysql> exit
- Rebuild slave from dump, this will again take some time
bring@mysql1qa:/tmp$ mysql -uroot -p < /tmp/dbdump.db
Enter password:
- Now login to mysql again and start slave and check status
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: exdap211.xyz.in
Master_User: slave_sec
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 62088
Relay_Log_File: mysqld-relay-bin.000003
Relay_Log_Pos: 62251
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 62088
Relay_Log_Space: 62425
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 100412
Master_UUID: f20637be-3530-11e5-a860-005056a05b60
Master_Info_File: /var/db/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
27 Jan 2016
These steps would help in resolving Mysql database replication issues arising out of duplicate entry in slave database.
Step-by-step guide
-
SSH to slave DB system. Example: ssh <user>@<server> (could be different server)
-
Connect to mysql. Example: mysql -u root -p. Note: It will prompt for password.
-
Check slave status. Example: show slave status\G. You might see something like this
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: <master-server.com>
Master_User: slave_sec
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000038
Read_Master_Log_Pos: 865774015
Relay_Log_File: mysqld-relay-bin.000021
Relay_Log_Pos: 25092925
Relay_Master_Log_File: mysql-bin.000038
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1062
Last_Error: Error 'Duplicate entry '6213' for key 'PRIMARY'' on query. Default database: 'testdb'. Query: 'insert into orders (created_at, user_id, receiver_id) values ('2016-01-12 08:48:09', null, null)'
Skip_Counter: 0
Exec_Master_Log_Pos: 817513358
Relay_Log_Space: 73353756
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1062
Last_SQL_Error: Error 'Duplicate entry '6213' for key 'PRIMARY'' on query. Default database: 'testdb'. Query: 'insert into orders (created_at, user_id, receiver_id) values ('2016-01-12 08:48:09', null, null)'
Replicate_Ignore_Server_Ids:
Master_Server_Id: 100412
Master_UUID: f20637be-3530-11e5-a860-005056a05b60
Master_Info_File: /var/db/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 160113 07:58:25
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
Note: if slave status returns nothing Empty set (0.00 sec) then you might be on master.
-
Better resolve this and don’t skip as suggested at many places as your slave can go out of sync with master and create problems in future.
This might be happening because some application might be pointing to slave DB instead of master, so best practice is to have slave as read only so that applications do not accidentally write to slave DB. Try to locate duplicate entry and delete that entry from slave DB.
-
Once you have deleted the old entry then execute stop slave and then start slave on slave DB.
-
Most probably replication will start again and come back to normal. If it gets stuck again for same error for some other record then repeat same steps. In case there are many such entries then better to reset slave and start replication again. Check slave status again to confirm.
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: <master-server.com>
Master_User: slave_sec
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000038
Read_Master_Log_Pos: 867336894
Relay_Log_File: mysqld-relay-bin.000021
Relay_Log_Pos: 57402302
Relay_Master_Log_File: mysql-bin.000038
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 849822735
Relay_Log_Space: 74916972
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 46579
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 100412
Master_UUID: f20637be-3530-11e5-a860-005056a05b60
Master_Info_File: /var/db/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
12 Nov 2015
I like Bower, simply because it does away with visiting individual sites to download latest java scripts. It’s been used mostly for java scripts though it is very generic and can handle any package. Bower is a fine package manager but it has its short comings. The main pain point for me is that it downloads the whole distribution with many files which are not needed for the project. That’s the reason most people dont check-in the bower_components in their repositories.
This is where Bower Installer comes to rescue. I came across this project around a month back and was really excited to use in my projects.
Before Bower Installer, my bower.json used to look like this.
{
"name": "explore-bower",
"version": "0.0.1",
"dependencies": {
"backbone": "latest",
"requirejs": "2.1.1",
"jquery": "2.1.4",
"bootstrap": "~3.3.5"
}
After Bower Installer, I had to make some changes to my bower.json
{
"name": "explore-bower",
"version": "0.0.1",
"dependencies": {
"backbone": "latest",
"requirejs": "2.1.1",
"jquery": "2.1.4",
"bootstrap": "~3.3.5"
},
"install": {
"path" : {
"css": "src/main/webapp/css",
"js": "src/main/webapp/js"
},
"sources" : {
"backbone" : [
"bower_components/backbone/backbone.js",
"bower_components/backbone/backbone-min.js"
],
"bootstrap" : [
"bower_components/bootstrap/dist/css/bootstrap.css",
"bower_components/bootstrap/dist/css/bootstrap.min.css",
"bower_components/bootstrap/dist/js/bootstrap.js"
]
}
}
}
install is the new section for bower installer- with
path you can define the destination folder individually for css and js folder.
- even without
sources section, bower installer can move the main files from bower componets to your defined destination folder, but with sources section you get greater control of what and how many files you want to move to your destination folder
Result: I found a very clean way to selectively choose the files that I need and move into my projects.
src
└── main
└── webapp
├── css
│ └── bootstrap
│ ├── bootstrap.css
│ └── bootstrap.min.css
└── js
├── backbone
│ ├── backbone-min.js
│ └── backbone.js
├── bootstrap
│ └── bootstrap.js
├── jquery
│ └── jquery.js
├── requirejs
│ └── require.js
└── underscore
└── underscore.js
The documentation of this project is very good, for complete reference and usage:
https://github.com/blittle/bower-installer